Method and device for measuring structural vibration of computer vision

Abstract

The invention discloses a structural vibration measurement method and device based on computer vision. The method comprises the following steps: converting a structural vibration video of a to-be-measured structure into an image sequence; selecting a target area in a first frame of the image sequence, and expanding the range of the target area according to the amplification factor to obtain a search area; taking the next frame as a current frame k, performing convolution on the filter mapping and the feature mapping in the search area of the current frame k to obtain response mapping in the search area, and determining a structure area tracked by the current frame k according to the maximum value of the response mapping in the search area; updating a search area and filter mapping according to the tracked structure area; sub-pixel image matching is used in the tracked structure area, structure vibration information of sub-pixel precision is obtained through calculation, and the step of taking the next frame as the current frame k is returned until all frames of the image sequence are processed. According to the embodiment of the invention, the influence of an extreme environment is avoided while the vibration of the video structure is identified.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of video image processing technology, and in particular to a computer vision method for measuring structural vibration. Background Art

[0002] Traditional structural displacement measurement methods usually rely on installing wired or wireless sensors on the structure, which directly measure the displacement, acceleration and other parameters of the structure. However, this may change the dynamic characteristics of the structure due to the addition of additional mass, especially on lightweight structures. For large structures, the installation process is not only time-consuming, but also consumes a lot of manpower and financial resources. More importantly, this type of sensor can only measure the response of a few specific points and cannot perform comprehensive measurements, which limits its application in high-resolution finite element model updates and associations.

[0003] In order to solve these problems, a computer vision-based structural vibration measurement method has emerged. The computer vision-based structural vibration measurement method uses imaging devices such as cameras to collect the dynamic response of the structure under the action of external forces, and extracts the displacement, velocity, acceleration and other vibration information of the structure through image processing and computational analysis. However, the computer vision-based structural measurement technology is highly dependent on the lighting conditions at the measurement site. Therefore, its accuracy and robustness may be affected by extreme environmental conditions caused by factors such as darkness, noise or blur. Therefore, there is an urgent need for a computer vision-based structural vibration measurement method that can achieve video structural vibration recognition while avoiding the influence of extreme environments. Summary of the invention

[0004] The present application provides a computer vision structural vibration measurement method and device, which can avoid the influence of extreme environments while identifying video structural vibration.

[0005] The embodiment of the present invention provides a computer vision structural vibration measurement method, comprising:

[0006] Convert the structural vibration video of the structure to be tested into an image sequence;

[0007] A target area is selected in the first frame of the image sequence, and the target area is enlarged according to the magnification factor to obtain a search area;

[0008] The next frame is taken as the current frame k, and the filter map in the search area of ​​the current frame k is convolved with the feature map to obtain the response map in the search area, and the structure area tracked by the current frame k is determined according to the maximum value of the response map in the search area;

[0009] Update the search area and filter map according to the tracked structure area;

[0010] Sub-pixel image matching is used in the tracked structure area to calculate and obtain structural vibration information with sub-pixel accuracy, and the next frame is returned as the current frame k until all frames of the image sequence are processed.

[0011] In an exemplary embodiment, converting the structural vibration video of the structure to be measured into an image sequence includes:

[0012] Converting the original structural vibration video of the structure to be tested into an initial image sequence;

[0013] The converted initial image sequence is converted from the RGB color space to the YIQ color space to obtain the image sequence.

[0014] In an exemplary embodiment, after selecting the target region of interest in the first frame of the image sequence, the method further includes:

[0015] The image features of the target area are extracted, and the extracted multiple features are fused; an initial filter mapping F0 is generated according to the fused feature representation.

[0016] In an exemplary embodiment, convolving the filter map in the search area of ​​the current frame k with the feature map to obtain a response map in the search area, and determining the structure area tracked by the current frame k according to the maximum value of the response map in the search area, includes:

[0017] For each frame starting from the second frame in the image sequence, extract grayscale feature information from the search area, and calculate a feature map within the search area of ​​the current frame image;

[0018] Convolving the filter map in the search area of ​​the current frame with the feature map to obtain a response map in the search area;

[0019] The structure area tracked by the current frame is determined according to the obtained maximum value of the response mapping within the search area and the size of the target area.

[0020] In an exemplary embodiment, updating the search area and the filter map according to the tracked structure area includes:

[0021] Expanding the tracked structural region according to the amplification factor to obtain a new region to update the search region;

[0022] Crop a sub-image within the search area according to the updated search area;

[0023] Resize the sub-image to M×N, extract grayscale features from the sub-image, and generate a feature map X of size M×N t+1 ;

[0024] Update the filter mapping according to the following formula to obtain a new filter mapping F t+2 :

[0025]

[0026] Among them, α k represents the weight of the k-th frame image; X k The matrix representation of the characteristic values ​​of the pixels in the current search area in the k-th frame image is the feature map; R k represents the response mapping of the kth frame; E(F t+2 ) represents a filter F t+2 The objective function of .

[0027] In an exemplary embodiment, when the filter mapping is updated, the method further includes:

[0028] The filter map is updated by incorporating structural spatial information and structural saliency information.

[0029] In an exemplary embodiment, the filter mapping is updated using the following formula to obtain a new filter mapping F t+2 :

[0030]

[0031] Among them, W represents the regularized weight mapping; ⊙ represents the corresponding product of matrix elements.

[0032] In an exemplary embodiment, when the filter mapping is updated, the method further includes:

[0033] The filter map is updated by embedding the total variation model. In this case, W in the subsequent frame is updated according to the following formula: t+1 Perform total variation optimization to solve:

[0034]

[0035] Among them, G t+1 W t+1 The intermediate variable, λ represents the regularization parameter, and u represents DW t+1 The intermediate variable of D represents the first-order finite difference operator along the two directions of the image; the constraint condition is: u = DW t+1 .

[0036] In an exemplary embodiment, the sub-pixel image matching is used in the tracked structural area to calculate and obtain structural vibration information with sub-pixel accuracy, including:

[0037] Fourier transform is used to calculate the mutual correlation coefficient between the structural regions tracked in the current frame and the previous frame and analyze the correlation peak position, and an initial estimation is performed based on the correlation peak to obtain the structural vibration information.

[0038] In an exemplary embodiment, after estimating the structural vibration information, the method further includes:

[0039] Selecting a small search area in the current frame according to the initial estimation, and upsampling the selected search area;

[0040] In the upsampled image, the cross-correlation coefficient is recalculated to find the location where the response is maximum;

[0041] In the cross-correlation result after upsampling, estimation is performed according to the determined correlation peak after upsampling to obtain the structural vibration information.

[0042] An embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the computer vision structural vibration measurement methods described above.

[0043] An embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: used to execute the steps of any of the above-mentioned computer vision structural vibration measurement methods.

[0044] The embodiment of the present application also provides a computer vision structural vibration measurement device, including: a conversion module, a processing module, a determination module, an update module, and a calculation module; wherein,

[0045] A conversion module, used for converting the structural vibration video of the structure to be tested into an image sequence;

[0046] A processing module, used for selecting a target area in the first frame of the image sequence, and expanding the target area according to the magnification factor to obtain a search area;

[0047] A determination module is used to take the next frame as the current frame k, convolve the filter map in the search area of ​​the current frame k with the feature map to obtain the response map in the search area, and determine the structure area tracked by the current frame k according to the maximum value of the response map in the search area;

[0048] An updating module, for updating the search area and the filter map according to the tracked structure area;

[0049] The calculation module is used to use sub-pixel image matching in the tracked structure area to calculate and obtain structural vibration information with sub-pixel accuracy, and return to the determination module until all frames of the image sequence are processed.

[0050] In an exemplary embodiment, the determining module is used to:

[0051] For each frame after the second frame in the image sequence, grayscale feature information is extracted from the search area, and a feature map within the search area of ​​the current frame image is calculated; a filter map within the search area of ​​the current frame is convolved with the feature map to obtain a response map within the search area; and a structural area tracked by the current frame is determined based on the maximum value of the response map within the search area and the size of the target area.

[0052] In an exemplary embodiment, the update module is used to:

[0053] The tracked structure region is expanded according to the magnification factor to obtain a new region to update the search region; a sub-image within the search region is cropped according to the updated search region; the size of the sub-image is adjusted to M×N, grayscale features are extracted from the sub-image, and a feature map X with a size of M×N is generated. t+1 ; Update the filter mapping according to the following formula to obtain a new filter mapping F t+2 :

[0054]

[0055] Among them, α k represents the weight of the k-th frame image; X k The matrix representation of the characteristic values ​​of the pixels in the current search area in the k-th frame image is the feature map; R k represents the response mapping of the kth frame; E(F t+2 ) represents a filter F t+2 The objective function of .

[0056] In an exemplary embodiment, the update module is further configured to:

[0057] Incorporate structural spatial information and structural saliency information to update the filter map;

[0058] And / or, embedding a total variation model to update the filter mapping.

[0059] In an exemplary embodiment, the computing module is used to:

[0060] Fourier transform is used to calculate the mutual correlation coefficient between the structural regions tracked in the current frame and the previous frame and analyze the correlation peak position, and an initial estimation is performed based on the correlation peak to obtain the structural vibration information.

[0061] In an exemplary embodiment, the computing module is further configured to:

[0062] After the structural vibration information is estimated, a small search area in the current frame is selected according to the initial estimation, and the selected search area is upsampled; in the upsampled image, the cross-correlation coefficient is recalculated; in the cross-correlation result after upsampling, an estimation is performed according to the determined correlation peak after upsampling to obtain the structural vibration information.

[0063] The computer vision structural vibration measurement method provided by the embodiment of the present application includes: converting the structural vibration video of the structure to be measured into an image sequence; selecting a target area in the first frame of the image sequence, and expanding the target area according to the amplification factor to obtain a search area; the next frame is used as the current frame k, and the filter map in the search area of ​​the current frame k is convolved with the feature map to obtain a response map in the search area, and the structural area tracked by the current frame k is determined according to the maximum value of the response map in the search area; the search area and the filter map are updated according to the tracked structural area; sub-pixel image matching is used in the tracked structural area to calculate and obtain structural vibration information with sub-pixel accuracy, and the next frame is returned as the step of the current frame k until all frames of the image sequence are processed. The embodiment of the present application avoids the influence of extreme environments while identifying the structural vibration of the video.

[0064] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0066] Figure 1 A schematic diagram of the flow chart of the structural vibration measurement method using computer vision in an embodiment of the present application;

[0067] Figure 2 A schematic diagram showing the process of the computer vision structural vibration measurement method in an embodiment of the present application;

[0068] FIG3( a ) is a schematic diagram of a structural photograph of a shaking table test arrangement for a cold-formed steel wall structure according to an embodiment of the present application;

[0069] FIG3( b ) is a top view of the test device of the cold-formed steel wall structure shaking table test arrangement in an embodiment of the present application;

[0070] FIG4( a ) is a schematic diagram of a first tracking result of a cold-bent steel wall structure in an insufficiently illuminated environment according to an embodiment of the present application;

[0071] FIG4( b ) is a schematic diagram of a second tracking result of a cold-bent steel wall structure in an insufficiently illuminated environment according to an embodiment of the present application;

[0072] FIG5( a ) is a schematic diagram of a first tracking result of a cold-formed steel wall structure in a noisy environment according to an embodiment of the present application;

[0073] FIG5( b ) is a schematic diagram of a second tracking result of a cold-formed steel wall structure in a noisy environment according to an embodiment of the present application;

[0074] FIG6( a ) is a schematic diagram of a first tracking result of a cold-formed steel wall structure in a fuzzy environment according to an embodiment of the present application;

[0075] FIG6( b ) is a schematic diagram of a second tracking result of a cold-formed steel wall structure in a fuzzy environment according to an embodiment of the present application;

[0076] FIG. 7( a ) is a schematic diagram of a first vibration measurement result of a cold-formed steel wall structure in an insufficiently illuminated environment according to an embodiment of the present application;

[0077] FIG. 7( b ) is a schematic diagram of a second vibration measurement result of a cold-formed steel wall structure in an insufficiently illuminated environment according to an embodiment of the present application;

[0078] FIG8( a ) is a schematic diagram of a first vibration measurement result of a cold-formed steel wall structure in a noise environment according to an embodiment of the present application;

[0079] FIG8( b ) is a schematic diagram of a second vibration measurement result of the cold-formed steel wall structure in a noisy environment according to an embodiment of the present application;

[0080] FIG9( a ) is a schematic diagram of a first vibration measurement result of a cold-formed steel wall structure in a fuzzy environment according to an embodiment of the present application;

[0081] FIG9( b ) is a schematic diagram of a second vibration measurement result of a cold-formed steel wall structure in a fuzzy environment according to an embodiment of the present application;

[0082] Fig.10 Schematic diagram of the composition structure of the computer vision structural vibration measurement device in the embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0084] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0085] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0086] It is understood that the terms "first" and "second" used in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0087] It can be understood that the “connection” in the following embodiments should be understood as “electrical connection”, “communication connection”, etc. if the connected circuits, modules, units, etc. have electrical signals or data transmission between each other.

[0088] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0089] Figure 1 The flowchart of the structural vibration measurement method of computer vision in the embodiment of the present application is shown in FIG. Figure 2 The process diagram shown may include:

[0090] Step 100: Convert the structural vibration video of the structure to be tested into an image sequence.

[0091] In one exemplary embodiment, step 100 may include:

[0092] Convert the original structural vibration video of the structure to be tested into an initial image sequence;

[0093] The converted initial image sequence is converted from the RGB color space to the YIQ color space to obtain the image sequence.

[0094] In an exemplary embodiment, the continuous vibration video is decomposed into an image sequence composed of frame-by-frame images so that each frame can be analyzed independently. In one embodiment, a video processing library (such as OpenCV) can be used to decompose the video frame by frame and save it as a separate image file.

[0095] In an exemplary embodiment, in the obtained image sequence, each frame of the image is usually represented in the RGB color space. For further processing, especially when processing color information or doing certain specific analysis (such as phase information extraction), the image needs to be converted from the RGB color space to the YIQ or other color space. The YIQ color space helps to separate color information and brightness information. In one embodiment, formula (1) can be used to convert the image sequence from the RGB color space to the YIQ color space:

[0096]

[0097] In formula (1), Y represents a brightness signal, I represents a color difference signal to which the human eye is sensitive, Q represents a color difference signal to which the human eye is not sensitive, R represents a red channel image signal, G represents a green channel image signal, and B represents a blue channel image signal.

[0098] Step 101: Select a target area in the first frame of the image sequence, and expand the target area according to the magnification factor to obtain a search area.

[0099] In an exemplary embodiment, in order to ensure that in subsequent frames of the image sequence, even if the structure to be tested moves or changes in the video, the search area can still cover the relevant structure, in an embodiment of the present application, the target area selected in the first frame image is expanded according to the magnification factor to serve as the search area of ​​the subsequent frame, and the magnification factor K≥1. The target area may be an area of ​​interest for the structure to be tested. For example, if the selected area of ​​interest for the structure to be tested is an area of ​​10×10 pixels and the magnification factor is 2, then the search area of ​​the subsequent frames may be expanded to 20×20 pixels. Among them, the area of ​​interest for the structure to be tested as the target area may be the structural part that you want to track and analyze in the image. The magnification factor can be pre-set according to the actual application scenario.

[0100] In an exemplary embodiment, step 101 may further include:

[0101] After selecting the target area of ​​interest in the first frame, the image features (such as texture, color histogram, gradient, etc.) of the target area can be extracted; the extracted multiple features are fused; and an initial filter map F0 is generated based on the fused feature representation.

[0102] In some embodiments, fusing the extracted multiple features may include, for example, superimposing different features, or performing weighting or dimensionality reduction processing through a more advanced algorithm to generate a comprehensive feature vector or feature map.

[0103] Filter Maps Filter maps used for target tracking can be generated by, but are not limited to, convolutional neural networks (CNNs). A filter map is actually a weight map used to find regions similar to the initial target region in subsequent image frames. The generated filter map can be viewed as a template or filter that determines which regions in the image best match the characteristics of the target by performing convolution or correlation operations with the image.

[0104] In the subsequent process, as time goes by and the number of frames increases, this filter map can be dynamically updated with each frame to adapt to the appearance changes of the structure of the target area.

[0105] Step 102: The next frame is taken as the current frame k, and the filter map in the search area of ​​the current frame k is convolved with the feature map to obtain the response map in the search area, and the structure area tracked by the current frame k is determined according to the maximum value of the response map in the search area.

[0106] In one exemplary embodiment, step 102 may include:

[0107] First, for each frame starting from the second frame in the image sequence (starting from frame t+1), grayscale feature information is extracted from the search area, and a feature map in the search area of ​​the current frame image is calculated. In one embodiment, for the current frame (starting from frame t+1), a sub-image in the search area can be cropped and the size of the sub-image can be adjusted to M×N, where t=0, 1, 2, ...; grayscale features are then extracted from the adjusted sub-image to generate a feature map C of size M×N t+1 ;

[0108] Next, the filter map in the search area of ​​the current frame is convolved with the feature map to obtain the response map in the search area. In one embodiment, the filter map F in the current frame (starting from frame t+1) is convolved with the feature map to obtain the response map in the search area. t+1 And the calculated feature map C t+1 Perform convolution and calculate the response map R t+1 The calculation formula can be expressed as: R t+1 =C t+1 *Ft+1 . Where * indicates convolution calculation. Response map R t+1 The size of is the same as the search area. Each pixel value in the response map represents the degree of match between that pixel location and the filter map.

[0109] Then, the structure area tracked by the current frame is determined based on the maximum value of the response map in the search area and the size of the target area. t+1 The maximum value of represents the point in the search area that best matches the filter map, and this maximum value usually corresponds to the center of the structure area tracked in the current frame. In other words, the response map R t+1 The maximum value of is the center, and according to the size of the target area, the structure area tracked in the current frame is constructed, that is, the tracking area O t+1 The size of the target area may generally include two dimensions: width and height. For example, the size of the target area may be 50×50 pixels, indicating a square area whose width and height are both 50 pixels.

[0110] Step 103: Update the search area and filter map according to the tracked structure area.

[0111] In an exemplary embodiment, updating the search area and the filter map according to the tracked structure area in step 103 may include:

[0112] First, the tracked structural area is expanded according to the amplification factor to obtain a new area to update the search area;

[0113] Then, the sub-image within the search area is cropped according to the updated search area;

[0114] After that, the sub-image is resized to M×N, and the grayscale features are extracted from the sub-image to generate a feature map X of size M×N. t+1 ;

[0115] Finally, the filter map is updated according to formula (2) to obtain a new filter map F t+2 :

[0116]

[0117] In formula (2), α k represents the weight of the k-th frame image; X k The matrix representation of the characteristic values ​​of the pixels in the current search area in the k-th frame image is the feature map; R k represents the response mapping of the kth frame. t+2 ) represents a filter F t+2Optimizing the objective function can enable the filter to track the target more accurately in subsequent frames.

[0118] In one embodiment, the tracked structure area O can be adjusted according to the magnification factor. t+1 Zoom in nearby and use the zoomed-in area as the new search area, and use the new search area to update the previous search area.

[0119] In an exemplary embodiment, when updating the filter map in step 103, the structural space information and the structural significance information can also be incorporated to update the filter map. In one embodiment, the structural space information and the structural significance information are incorporated into the update iterative optimization formula of the filter map, and the corresponding regularization weight map needs to be added to the formula (2). At this time, the filter map is updated using the formula (3) to obtain a new filter map F t+2 :

[0120]

[0121] In formula (3), W represents the regularized weight mapping; ⊙ represents the corresponding product of matrix elements.

[0122] In an exemplary embodiment, when updating the filter map in step 103, a total variation model may be embedded to update the filter map. The purpose of embedding the total variation model is to enhance the filter map F t+2 The accuracy and robustness of the regularized weight map W during the update process. In one embodiment, therefore, only W in the subsequent frame needs to be updated according to formula (4). t+1 Perform total variation optimization to solve:

[0123]

[0124] In formula (4), G t+1 W t+1 The intermediate variable, λ represents the regularization parameter, and u represents DW t+1 The intermediate variable is u = DW, and D represents the first-order finite difference operator along the two directions of the image. The constraint condition of formula (4) is: t+1 .

[0125] By incorporating structural spatial information and structural significance information, and / or embedding a total variation model, the structural vibration measurement method based on computer vision of a total variation dynamic correlation filter provided in the embodiment of the present application achieves enhanced accuracy and robustness in structural region tracking, and provides a guarantee for maintaining tracking stability even in extreme situations.

[0126] In the embodiment of the present application, the filter mapping is optimized and updated in each frame, which helps the filter adapt to changes in the environment, such as changes in illumination, noise fluctuations, etc., thereby improving the tracking accuracy and robustness of the target area, and avoiding the influence of extreme environments while identifying the vibration of the video structure. In particular, the iterative update after incorporating the spatial information of the structure, the saliency information, and the total variation model helps the filter adapt to changes in the environment, such as changes in illumination, noise fluctuations, etc., thereby further improving the tracking accuracy and robustness of the target area.

[0127] Step 104: Use sub-pixel image matching in the tracked structure area to calculate and obtain structural vibration information with sub-pixel accuracy, and return to step 102 until all frames of the image sequence are processed.

[0128] In one exemplary embodiment, step 104 may include:

[0129] The Fourier transform is used to calculate the cross-correlation coefficient between the structural area tracked in the current frame and the first frame and analyze the correlation peak position. An initial estimate is made based on the correlation peak to obtain the structural vibration information. The initial estimate is the pixel-level displacement estimate of the structural area obtained through the preliminary cross-correlation calculation.

[0130] In one embodiment, the Fourier transform is used to calculate the cross-correlation coefficient between the structural regions tracked in the current frame and the first frame, and an initial estimate is made from the correlation peak to obtain the structural vibration. The calculation is shown in formula (5):

[0131]

[0132] In formula (5), O t (x, y) represents the structure area tracked in the current frame t, located at the coordinates (x, y); O1(x, y) represents the structure area tracked in the first frame, located at the coordinates (x, y); r(x0, y0) represents the tracking area O tracked in the current frame t The cross-correlation coefficient between (x, y) and the tracking area O1(x, y) in the first frame is used to quantify the similarity between the two frames at a given displacement (x0, y0). It means finding the displacement that minimizes the error among all possible (x0, y0) displacement combinations, which means we hope to find a displacement (x0, y0) such that the structural area of ​​the current frame and the first frame have the highest similarity under this displacement.

[0133] In an exemplary embodiment, in order to further improve the accuracy of vibration information, after estimating the structural vibration information, the following may also be included:

[0134] A small search area in the current frame is selected based on the initial estimation, and the selected search area is upsampled; in the upsampled image, the cross-correlation coefficient is recalculated, so that a search can be performed on a higher resolution image to find the position with the largest response; in the upsampled cross-correlation result, an estimation is performed based on the determined correlation peak after upsampling to obtain structural vibration information, where the position of the determined correlation peak after upsampling corresponds to the structural area displacement with sub-pixel accuracy.

[0135] In one embodiment, recalculating the cross-correlation coefficient may include:

[0136] The mutual correlation coefficient within a preset pixel range, such as 1.5×1.5 pixel range, around the initially estimated structure can be calculated based on the upsampling factor κ. In this way, the sub-pixel resolution within the 1 / κ pixel range can be achieved by finding the maximum value within the 1.5κ×1.5κ pixel range, that is, by finding the maximum value of the mutual correlation coefficient within the amplified 1.5κ×1.5κ pixel range, the displacement of the structure can be located at the sub-pixel level. The sub-pixel resolution within the 1 / κ pixel range means that search and matching can be performed at a finer scale (such as 1 / 2 pixel or 1 / 4 pixel), thereby obtaining a higher precision displacement estimate.

[0137] In the embodiment of the present application, by performing a fine search on a small area around the initial estimated position, the computational burden of performing a large-scale search on the entire image is avoided, and the possibility of being affected by extreme conditions (such as large-scale lighting changes, background interference, etc.) is also reduced. At high resolution, the system can better cope with subtle environmental changes, so that the tracking stability can be maintained in extreme situations.

[0138] In an exemplary embodiment, when updating the filter map, the structural space information and the structural saliency information are incorporated into the filter map to update the filter map, the implementation of incorporating the structural space information and the structural saliency information into the optimization formula may include one of the following:

[0139] In order to incorporate structural spatial information, we can use formula (6) to construct W in formula (3) as a spatially aware regularized weight mapping W SA (x, y), formula (6) is expressed as:

[0140]

[0141] In formula (6), x = 1, 2, ..., M and y = 1, 2, ..., N are the coordinate indices of the sub-image, that is, the position of each pixel in the M × N sub-image; a>0, b>0 are the correlation coefficients used to determine W SA The maximum and minimum values ​​of .

[0142] In order to incorporate structural saliency information, the saliency map S in the frame can be obtained by saliency detection calculation. t+1 , and then use formula (7) to convert S t+1 Normalized to S t ′ +1 (x,y):

[0143]

[0144] In formula (7), Max S and Min S Respectively represent S t+1 The maximum and minimum values ​​of .

[0145] In order to incorporate the structural space information and structural saliency information into the optimization problem, the regularized weight mapping W = W t+1 The dynamic update of can be achieved by formula (8):

[0146]

[0147] In formula (8), β is the influencing parameter, and the structural area Ω str and the unstructured region Ω non It can be expressed as shown in formula (9):

[0148]

[0149] In formula (9), ζ>0 is a threshold used to distinguish the structural region Ω str and the unstructured region Ω non , μ str , μ non S t+1 In Ω str ,Ω non The average value in .

[0150] Finally, the filter update iterative optimization can be obtained by formula (10):

[0151]

[0152] In formula (10), δ is the partial derivative, W S ′ A It is W SA A negative number.

[0153] In one exemplary embodiment, to enhance the filter mapping F t+2 The accuracy and robustness of the regularized weight map W during the update process can be further embedded in the total variation model, that is, W in the subsequent frame t+1Perform total variation optimization to solve the problem. The optimization is shown in formula (4):

[0154] In order to solve formula (4), it can be achieved through formulas (11), (12), and (13):

[0155]

[0156] In formulas (11), (12), and (13), u is DW t+1 The intermediate variable, γ is the regularization parameter, and the constraint ‖u-DW t+1 ‖ 2 related, j is the Lagrange multiplier, and the constraint u = DW t+1 τ2 is the number of optimization iterations. is the Fourier transform, D x is the component of D in the x direction, D y is the component of D in the y direction.

[0157] In some embodiments of the present application, structural spatial information and structural significance information are incorporated into the correlation filter and embedded in the total variation model; then, based on the structural area tracked in each frame, it is further optimized using sub-pixel template matching to achieve more accurate recognition of structural vibrations.

[0158] In an exemplary embodiment, the computer vision structural vibration measurement method based on total variation dynamic correlation filter provided in the embodiment of the present application can be evaluated from a video recognition example. In order to objectively test the technical effect of the computer vision structural vibration measurement method provided in the embodiment of the present application, a public structural vibration video is selected as an example. In this embodiment, in order to ensure the test of the algorithm stability, severe low light, noise and blur environment are pre-generated in all analyzed video signals.

[0159] The experimental arrangement of the shaking table test used is shown in Figure 3 (a) and Figure 3 (b). In this embodiment, the test video resolution is 654×368, the frame rate is 24Hz, and the number of frames is 1000. The purpose is to study the lateral vibration response of the cold-formed steel wall structure. In this embodiment, the cold-formed steel wall structure used for testing is a shear-gravity-gravity-shear wall. There is a pair of tie rods at the end of each shear wall segment. A specific fastener pattern is used on the steel plate. The structural position to be measured is shown in the white box in Figure 3 (a).

[0160] Figure 4(a) is a schematic diagram of the first tracking result of the cold-bent steel wall structure in an insufficiently illuminated environment (brightness reduced to 75) in an embodiment of the present application, and Figure 4(b) is a schematic diagram of the second tracking result of the cold-bent steel wall structure in an insufficiently illuminated environment (brightness reduced to 150) in an embodiment of the present application; Figure 5(a) is a schematic diagram of the first tracking result of the cold-bent steel wall structure in a noisy environment (noise standard deviation 100) in an embodiment of the present application, and Figure 5(b) is a schematic diagram of the second tracking result of the cold-bent steel wall structure in a noisy environment (noise standard deviation 150) in an embodiment of the present application; Figure 6(a) is a schematic diagram of the first tracking result of the cold-bent steel wall structure in a fuzzy environment (fuzzy standard deviation 10) in an embodiment of the present application, and Figure 6(b) is a schematic diagram of the second tracking result of the cold-bent steel wall structure in an fuzzy environment (fuzzy standard deviation 15) in an embodiment of the present application. As shown in the figure, in an environment with severe lack of lighting, severe noise and severe blur, the tracking result comes from the 600th frame. From the results of structure tracking using the structural vibration measurement method of computer vision provided in an embodiment of the present application, it can be found that despite the severe lack of lighting, severe noise covering the video information, and severe blur, the structural information can still be detected relatively accurately.

[0161] Figure 7(a) is a schematic diagram of the first vibration measurement result of the cold-bent steel wall structure in an insufficiently illuminated environment (brightness reduced to 75) in an embodiment of the present application, and Figure 7(b) is a schematic diagram of the second vibration measurement result of the cold-bent steel wall structure in an insufficiently illuminated environment (brightness reduced to 150) in an embodiment of the present application. The method proposed in the figure refers to the structural vibration measurement method of computer vision provided in an embodiment of the present application; Figure 8(a) is a schematic diagram of the first vibration measurement result of the cold-bent steel wall structure in a noisy environment (noise standard deviation 100) in an embodiment of the present application, and Figure 8(b) is a schematic diagram of the second vibration measurement result of the cold-bent steel wall structure in a noisy environment (noise standard deviation 150) in an embodiment of the present application; Figure 9(a) is a schematic diagram of the first vibration measurement result of the cold-bent steel wall structure in a fuzzy environment (fuzzy standard deviation 10) in an embodiment of the present application, and Figure 9(b) is a schematic diagram of the second vibration measurement result of the cold-bent steel wall structure in a fuzzy environment (fuzzy standard deviation 15) in an embodiment of the present application. As shown in the figure, similarly, under poor lighting conditions, severe noise and blur, from the recognition results of structural vibration using the structural vibration measurement method of computer vision provided in the embodiment of the present application, it can be seen that the measured structural vibration information is extremely close to the result recorded by the reference sensor, and can achieve 0.08 sub-pixel accuracy results.

[0162] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the computer vision structural vibration measurement methods described above.

[0163] The present application further provides a computer device, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: used to execute the steps of any of the above-mentioned computer vision structural vibration measurement methods.

[0164] Fig.10 Schematic diagram of the structure of the computer vision structural vibration measurement device in the embodiment of the present application. Fig.10 As shown, it may include: a conversion module, a processing module, a determination module, an update module, and a calculation module; wherein,

[0165] A conversion module, used for converting the structural vibration video of the structure to be tested into an image sequence;

[0166] A processing module, used for selecting a target area in the first frame of the image sequence, and expanding the target area according to the magnification factor to obtain a search area;

[0167] A determination module is used to take the next frame as the current frame k, convolve the filter map in the search area of ​​the current frame k with the feature map to obtain the response map in the search area, and determine the structure area tracked by the current frame k according to the maximum value of the response map in the search area;

[0168] An updating module, for updating the search area and the filter map according to the tracked structure area;

[0169] The calculation module is used to use sub-pixel image matching in the tracked structure area to calculate and obtain structural vibration information with sub-pixel accuracy, and return to the determination module until all frames of the image sequence are processed.

[0170] In an exemplary embodiment, the processing module is further configured to:

[0171] After selecting the target area of ​​interest in the first frame, the image features (such as texture, color histogram, gradient, etc.) of the target area are extracted; the extracted multiple features are fused; the fused feature representation is used to generate an initial filter map F0.

[0172] In one exemplary embodiment, the determination module may be configured to:

[0173] For each frame after the second frame in the image sequence (starting from frame t+1), the grayscale feature information is extracted from the search area, and the feature map within the search area of ​​the current frame image is calculated; the filter map within the search area of ​​the current frame is convolved with the feature map to obtain the response map within the search area; based on the maximum value of the response map within the search area and the size of the target area, the structure area tracked by the current frame is determined.

[0174] In one exemplary embodiment, the update module may be used to:

[0175] According to the magnification factor, the tracked structure area is expanded to obtain a new area to update the search area; according to the updated search area, a sub-image within the search area is cropped; the size of the sub-image is adjusted to M×N, grayscale features are extracted from the sub-image, and a feature map X with a size of M×N is generated t+1 ; Update the filter mapping according to formula (2) to obtain a new filter mapping F t+2 .

[0176] In an exemplary embodiment, the update module is further configured to:

[0177] Incorporate structural spatial information and structural saliency information to update the filter map;

[0178] And / or, embedding a total variation model to update the filter mapping.

[0179] In one exemplary embodiment, the computing module may be used to:

[0180] Fourier transform is used to calculate the cross-correlation coefficient between the structural regions tracked in the current frame and the previous frame and analyze the correlation peak position. An initial estimate is made based on the correlation peak to obtain the structural vibration information. The initial estimate is the pixel-level displacement estimate of the structural region obtained through preliminary cross-correlation calculation.

[0181] In an exemplary embodiment, the computing module may also be used to:

[0182] After estimating the structural vibration information, a small search area in the current frame is selected according to the initial estimation, and the selected search area is upsampled; in the upsampled image, the cross-correlation coefficient is recalculated; in the upsampled cross-correlation result, an estimation is performed based on the determined correlation peak after upsampling to obtain the structural vibration information.

[0183] The computer vision structural vibration measurement device provided in the embodiment of the present application avoids the influence of extreme environments while identifying the structural vibration of the video.

[0184] Although the embodiments disclosed in this application are as above, the contents described are only embodiments adopted to facilitate understanding of this application and are not intended to limit this application. Any technician in the field to which this application belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in this application, but the scope of patent protection of this application shall still be based on the scope defined in the attached claims.

Claims

1. A computer vision structural vibration measurement method, characterized in that: include: Convert the structural vibration video of the structure to be tested into an image sequence; A target area is selected in the first frame of the image sequence, and the target area is enlarged according to the magnification factor to obtain a search area; The next frame is taken as the current frame k, and the filter map in the search area of ​​the current frame k is convolved with the feature map to obtain the response map in the search area, and the structure area tracked by the current frame k is determined according to the maximum value of the response map in the search area; Update the search area and filter map according to the tracked structure area; Sub-pixel image matching is used in the tracked structure area to calculate and obtain structural vibration information with sub-pixel accuracy, and the next frame is returned as the current frame k until all frames of the image sequence are processed.

2. The structural vibration measurement method according to claim 1, wherein: The step of converting the structural vibration video of the structure to be tested into an image sequence comprises: Converting the original structural vibration video of the structure to be tested into an initial image sequence; The converted initial image sequence is converted from the RGB color space to the YIQ color space to obtain the image sequence.

3. The structural vibration measurement method according to claim 1, after selecting a target area of ​​interest in the first frame of the image sequence, further comprising: Extracting image features of the target area and fusing the extracted multiple features; Based on the fused feature representation, an initial filter map F0 is generated.

4. The structural vibration measurement method according to claim 1, wherein: The filter map in the search area of ​​the current frame k is convolved with the feature map to obtain a response map in the search area, and the structure area tracked by the current frame k is determined according to the maximum value of the response map in the search area, including: For each frame starting from the second frame in the image sequence, extract grayscale feature information from the search area, and calculate a feature map within the search area of ​​the current frame image; Convolving the filter map in the search area of ​​the current frame with the feature map to obtain a response map in the search area; The structure area tracked by the current frame is determined according to the obtained maximum value of the response mapping within the search area and the size of the target area.

5. The structural vibration measurement method according to claim 1, wherein: The updating of the search area and the filter map according to the tracked structure area comprises: Expanding the tracked structural region according to the amplification factor to obtain a new region to update the search region; Crop a sub-image within the search area according to the updated search area; Resize the sub-image to M×N, extract grayscale features from the sub-image, and generate a feature map X of size M×N t+1 ; Update the filter mapping according to the following formula to obtain a new filter mapping F t+2 : Among them, α k represents the weight of the k-th frame image; X k The matrix representation of the characteristic values ​​of the pixels in the current search area in the k-th frame image is the feature map; R k represents the response mapping of the kth frame; E(F t+2 ) represents a filter F t+2 The objective function of .

6. The structural vibration measurement method according to claim 5, wherein: When the filter mapping is updated, it also includes: The filter map is updated by incorporating structural spatial information and structural saliency information.

7. The structural vibration measurement method according to claim 6, wherein: The filter mapping is updated using the following formula to obtain a new filter mapping F t+2 : Among them, W represents the regularized weight mapping; ⊙ represents the corresponding product of matrix elements.

8. The structural vibration measurement method according to claim 5 or 7, wherein: When the filter mapping is updated, it also includes: The filter map is updated by embedding the total variation model. In this case, W in the subsequent frame is updated according to the following formula: t+1 Perform total variation optimization to solve: subject to u=DW t+1 Among them, G t+1 W t+1 The intermediate variable, λ represents the regularization parameter, and u represents DW t+1 The intermediate variable of D represents the first-order finite difference operator along the two directions of the image; the constraint condition is: u = DW t+1 .

9. The structural vibration measurement method according to claim 1, wherein: The sub-pixel image matching is used in the tracked structural area to calculate and obtain structural vibration information with sub-pixel accuracy, including: Fourier transform is used to calculate the mutual correlation coefficient between the structural regions tracked in the current frame and the previous frame and analyze the correlation peak position, and an initial estimation is performed based on the correlation peak to obtain the structural vibration information.

10. The structural vibration measurement method according to claim 9, after estimating the structural vibration information, further comprising: Selecting a small search area in the current frame according to the initial estimation, and upsampling the selected search area; In the upsampled image, the cross-correlation coefficient is recalculated to find the location where the response is maximum; In the cross-correlation result after upsampling, estimation is performed according to the determined correlation peak after upsampling to obtain the structural vibration information.

11. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the computer vision structural vibration measurement method according to any one of claims 1 to 10.

12. A computer device comprising a memory and a processor, wherein: The memory stores the following instructions executable by the processor: used to execute the steps of the computer vision structural vibration measurement method as described in any one of claims 1 to 10.

13. A computer vision structural vibration measurement device, characterized in that: include: A conversion module, a processing module, a determination module, an update module, and a calculation module; wherein, A conversion module, used for converting the structural vibration video of the structure to be tested into an image sequence; A processing module, used for selecting a target area in the first frame of the image sequence, and expanding the target area according to the magnification factor to obtain a search area; A determination module is used to take the next frame as the current frame k, convolve the filter map in the search area of ​​the current frame k with the feature map to obtain the response map in the search area, and determine the structure area tracked by the current frame k according to the maximum value of the response map in the search area; An updating module, for updating the search area and the filter map according to the tracked structure area; The calculation module is used to use sub-pixel image matching in the tracked structure area to calculate and obtain structural vibration information with sub-pixel accuracy, and return to the determination module until all frames of the image sequence are processed.

14. The structural vibration measuring device according to claim 13, wherein the determining module is used for: For each frame after the second frame in the image sequence, grayscale feature information is extracted from the search area, and a feature map within the search area of ​​the current frame image is calculated; a filter map within the search area of ​​the current frame is convolved with the feature map to obtain a response map within the search area; and a structural area tracked by the current frame is determined based on the maximum value of the response map within the search area and the size of the target area.

15. The structural vibration measuring device according to claim 13, wherein the updating module is used for: Expanding the tracked structure region according to the magnification factor to obtain a new region to update the search region; cutting out a sub-image within the search region according to the updated search region; Resize the sub-image to M×N, extract grayscale features from the sub-image, and generate a feature map X of size M×N t+1 ; Update the filter mapping according to the following formula to obtain a new filter mapping F t+2 : in, α k represents the weight of the k-th frame image; X k The matrix representation of the eigenvalues ​​of the pixels in the current search area in the k-th frame image is the feature map; R k represents the response mapping of the kth frame; E(F t+2 ) represents a filter F t+2 The objective function of .

16. The structural vibration measuring device according to claim 15, wherein the updating module is further used for: Incorporate structural spatial information and structural saliency information to update the filter map; And / or, embedding a total variation model to update the filter mapping.

17. The structural vibration measuring device according to claim 13, wherein the calculation module is used for: Fourier transform is used to calculate the mutual correlation coefficient between the structural regions tracked in the current frame and the previous frame and analyze the correlation peak position, and an initial estimation is performed based on the correlation peak to obtain the structural vibration information.

18. The structural vibration measuring device according to claim 17, wherein the calculation module is further used for: After the structural vibration information is estimated, a small search area in the current frame is selected according to the initial estimation, and the selected search area is upsampled; in the upsampled image, the cross-correlation coefficient is recalculated; in the cross-correlation result after upsampling, an estimation is performed according to the determined correlation peak after upsampling to obtain the structural vibration information.

Images