Structural vibration displacement identification method based on diamond search and improved optical flow method

Abstract

The present invention discloses a structural vibration displacement identification method based on diamond search and improved optical flow method. Determine the region of interest and origin coordinates of the first video image frame in a video image frame sequence; for each video image frame except the first video image frame, use the origin coordinates of the final displacement determination position of the previous video image frame as the search starting point, use the path determined by the large diamond search mode and the small diamond search mode as the search path, use the area size of the region of interest as the calculation area of ​​each path point, search, calculate the two-dimensional correlation coefficient between the calculation area and the region of interest at each search path point; obtain the first calculation area corresponding to the maximum value of all two-dimensional correlation coefficients; determine the initial displacement; construct an optical flow gradient equation group based on the first calculation area and the region of interest, use a preset operator to calculate, determine the fine displacement; and finally determine the final displacement. This solution realizes the precise tracking of structural displacement.

Description

Technical Field

[0001] The present invention relates to the technical field of structural health monitoring, and in particular to a structural vibration displacement identification method based on diamond search and improved optical flow method. Background Art

[0002] With the continuous development of science and technology, vision-based motion tracking technology has been widely used in various fields, such as machine vision, target tracking, human behavior analysis, structural health monitoring, etc. Among them, computer vision-based structural displacement monitoring technology has been widely used in recent years due to its safety, convenience, and practicality.

[0003] However, existing vision-based displacement tracking algorithms, such as the Lucas-Kanade optical flow method, have high calculation accuracy under small motion conditions. However, when the target moves greatly, the displacement calculation results cannot be updated in real time due to the inability to update the region of interest, which cannot meet the needs of high-precision measurement. Summary of the invention

[0004] In order to solve the technical problem that the existing vision-based displacement tracking algorithm cannot meet the high-precision measurement requirements, an embodiment of the present invention provides a structural vibration displacement identification method based on diamond search and improved optical flow method.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] The embodiment of the present invention provides a structural vibration displacement identification method based on diamond search and improved optical flow method, the method comprising: obtaining a video image frame sequence to be processed; the video image frame sequence is obtained by performing frame processing on the displacement video of the target structure in time sequence; determining an area of ​​interest of the first video image frame in the video image frame sequence, the first video image frame being the first video image frame in the video image frame sequence, and the area of ​​interest being the area where the target structure is located in the first video image frame; based on the area of ​​interest, determining the pixel coordinates of the upper left corner point of the area of ​​interest as the origin coordinates of the area of ​​interest; for each of the remaining video image frames except the first video image frame in the video image frame sequence, taking the origin coordinates of the position determined by the final displacement of the previous video image frame as the search starting point, taking the path determined by the large diamond search mode and the small diamond search mode as the search path, and taking the area size of the area of ​​interest as each A calculation area of ​​a path point is searched in a large diamond search and a small diamond search, and a two-dimensional correlation coefficient between the calculation area and the region of interest in the first video image frame is calculated at each search path point; the two-dimensional correlation coefficients of all search path points are obtained, and a first calculation area corresponding to the maximum value of the two-dimensional correlation coefficients is determined; according to the position coordinates of the first calculation area and the position coordinates of the region of interest in the first video image frame, an initial displacement of the target structure in each video image frame is determined; for each video image frame except the first video image frame in the video image frame sequence, an optical flow gradient equation group is constructed based on the first calculation area in each video image frame and the region of interest in the first video image frame, and a preset operator is used to calculate the optical flow gradient equation group to determine the fine displacement of the target structure in each video image frame; the initial displacement and the fine displacement are added to determine the final displacement of the target structure in each video image frame.

[0007] In one embodiment, the path determined based on the large diamond search mode and the small diamond search mode is: obtaining a first search path of the large diamond search mode, taking the origin coordinates of the position determined by the final displacement of the previous video image frame as the search starting point, and performing a large diamond search according to the first search path; for each search point on the first search path, comparing the two-dimensional correlation coefficient calculated for each search point with the two-dimensional correlation coefficient calculated for the previous search point, if the two-dimensional correlation coefficient calculated for the current search point is greater than the two-dimensional correlation coefficient calculated for the previous search point, then not performing a small diamond search for the previous search point, and continuing to perform a large diamond search for the next search point according to the first search path; if the two-dimensional correlation coefficient calculated for the current search point is not greater than the two-dimensional correlation coefficient calculated for the previous search point, then obtaining a second search path of the small diamond search mode, taking the previous search point as the search starting point, and performing a small diamond search according to the second search path; the obtaining of the two-dimensional correlation coefficients of all search path points includes: obtaining the two-dimensional correlation coefficients calculated for all search points in the large diamond search mode and the two-dimensional correlation coefficients calculated for all search points in the small diamond search mode.

[0008] In one embodiment, the first search path of the large diamond search pattern is: a path formed by each point in sequence in the following calculation formula is the first search path of the large diamond search pattern: Calculation formula (1).

[0009] In one embodiment, the second search path of the small diamond search pattern is: a path formed by each point in sequence in the following calculation formula is the second search path of the small diamond search pattern:

[0010] Calculation formula (2).

[0011] In one embodiment, the calculating of the two-dimensional correlation coefficient between the calculation area and the region of interest in the first video image frame at each search path point includes: calculating the two-dimensional correlation coefficient between the calculation area and the region of interest in the first video image frame using the following calculation formula:

[0012] Calculation formula (3)

[0013] in, is the two-dimensional correlation coefficient, A is the calculation area, B is the region of interest in the first video image frame, m and n are the horizontal and vertical pixel dimensions of the image, respectively, is the mean pixel intensity of the calculation area, is the mean pixel intensity of the region of interest in the first video image frame.

[0014] In one embodiment, constructing an optical flow gradient equation group based on the first calculation area in each video image frame and the region of interest in the first video image frame includes: the constructed optical flow gradient equation group is shown in the following calculation formula:

[0015] Calculation formula (4)

[0016] in, is the gray value of a pixel among the n pixels in the region of interest, represents the horizontal coordinate of the pixel, represents the vertical coordinate of the pixel, Indicates time, is the gray value of the corresponding pixel in the first calculation area, and Respectively represent the small displacement of the first calculation area relative to the region of interest in the horizontal and vertical directions, represents the time increment, , Represents the higher level of Peano, , , , Respectively represent the optical flow gradient of the image in the horizontal, vertical and time directions, and use , , express , , , , is the optical flow gradient of each pixel.

[0017] In one embodiment, the optical flow gradient equations are calculated using a preset operator to determine the fine displacement of the target structure in each video image frame, including: calculating the horizontal optical flow gradient using the following preset operator and vertical optical flow gradient :

[0018] Calculation formula (5)

[0019] Calculation formula (6)

[0020] in, Representative i frame, , Represents the first pixel matrix of the calculation area j Row, No. k List, Represents the image sequence i The calculation area in the frame j Row, No. k The pixel grayscale value of the column; the horizontal optical flow gradient to be calculated and vertical optical flow gradient Substitute into the optical flow gradient equation group, and use the least square method and matrix inverse operation to calculate and obtain the fine displacement of the target structure in the horizontal and vertical directions in each video image frame.

[0021] The structural vibration displacement identification method based on diamond search and improved optical flow method provided by the embodiment of the present invention obtains a video image frame sequence to be processed; the video image frame sequence is obtained by performing frame processing on the displacement video of the target structure in time sequence; the region of interest of the first video image frame in the video image frame sequence is determined, the first video image frame is the first video image frame in the video image frame sequence, and the region of interest is the region where the target structure is located in the first video image frame; based on the region of interest, the pixel coordinates of the upper left corner point of the region of interest are determined as the origin coordinates of the region of interest; for each of the remaining video image frames except the first video image frame in the video image frame sequence, the origin coordinates of the final displacement determined position of the previous video image frame are used as the search starting point, the path determined by the large diamond search mode and the small diamond search mode is used as the search path, and the area size of the region of interest is used as the coordinate of each path point The calculation area is used to perform large diamond search and small diamond search, and the two-dimensional correlation coefficient between the calculation area and the region of interest in the first video image frame is calculated at each search path point; the two-dimensional correlation coefficients of all search path points are obtained, and the first calculation area corresponding to the maximum value of the two-dimensional correlation coefficients is determined; according to the position coordinates of the first calculation area and the position coordinates of the region of interest in the first video image frame, the initial displacement of the target structure in each video image frame is determined; for each video image frame except the first video image frame in the video image frame sequence, an optical flow gradient equation group is constructed based on the first calculation area in each video image frame and the region of interest in the first video image frame, and the optical flow gradient equation group is calculated using a preset operator to determine the fine displacement of the target structure in each video image frame; the initial displacement and the fine displacement are added to determine the final displacement of the target structure in each video image frame. The scheme provided by the present invention effectively integrates the scale characteristics of the diamond search algorithm and the target-free characteristics of the optical flow method, realizes the accurate tracking of the structural displacement, and effectively avoids the defect that the traditional optical flow method is limited to the assumption of small displacement measurement and cannot update the region of interest in real time to track large displacement. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a method for identifying structural vibration displacement based on diamond search and improved optical flow method according to an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of search points in a large diamond search mode according to an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of search points in a small diamond search mode according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of optical flow calculation according to an embodiment of the present invention;

[0026] Figure 5 Schematic diagram of the comparison of calculation errors of different operators in the embodiment of the present invention;

[0027] FIG6 (a) is a schematic diagram of a field test scene of an embodiment of the present invention from a perspective facing the cable;

[0028] FIG6 ( b ) is a schematic diagram of a side instrument view of a field test scene according to an embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram of cable and laser point displacement results extracted by a drone according to an embodiment of the present invention;

[0030] Figure 8 This is a schematic diagram comparing the displacement results extracted by the drone and the displacement results extracted by the fixed camera according to an embodiment of the present invention;

[0031] Fig. 9 This is a schematic diagram of the result of the UAV identifying the cable frequency according to an embodiment of the present invention;

[0032] Fig.10 This is a schematic diagram of the result of the accelerometer identifying the cable frequency according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0034] The embodiment of the present invention provides a structural vibration displacement identification method based on diamond search and improved optical flow method, such as Figure 1 As shown, the method includes:

[0035] Step 101: Acquire a video image frame sequence to be processed; the video image frame sequence is obtained by performing frame processing on the displacement video of the target structure in time sequence;

[0036] Step 102: determining a region of interest of a first video image frame in the video image frame sequence, wherein the first video image frame is the first video image frame in the video image frame sequence, and the region of interest is a region where a target structure is located in the first video image frame; based on the region of interest, determining pixel coordinates of an upper left corner point of the region of interest as origin coordinates of the region of interest;

[0037] Step 103: for each of the remaining video image frames except the first video image frame in the video image frame sequence, the coordinates of the origin of the final displacement position of the previous video image frame are used as the search starting point, the path determined by the large diamond search mode and the small diamond search mode is used as the search path, and the area size of the region of interest is used as the calculation area of ​​each path point, and a large diamond search and a small diamond search are performed to calculate the two-dimensional correlation coefficient between the calculation area and the region of interest in the first video image frame at each search path point; the two-dimensional correlation coefficients of all search path points are obtained, and the first calculation area corresponding to the maximum value of the two-dimensional correlation coefficients is determined; according to the position coordinates of the first calculation area and the position coordinates of the region of interest in the first video image frame, the initial displacement of the target structure in each video image frame is determined;

[0038] Step 104: for each of the remaining video image frames except the first video image frame in the video image frame sequence, construct an optical flow gradient equation group based on the first calculation region in each of the video image frames and the region of interest in the first video image frame, and calculate the optical flow gradient equation group using a preset operator to determine the fine displacement of the target structure in each of the video image frames;

[0039] Step 105: Add the initial displacement and the refined displacement to determine the final displacement of the target structure in each video image frame.

[0040] The diamond search algorithm is a fast search algorithm based on block matching. It searches in large and small diamond areas of different scales to find the optimal motion vector, thereby improving the search efficiency and accuracy.

[0041] Optical flow is a technique used to estimate the motion of objects in an image sequence. It calculates the instantaneous speed of pixels, i.e., the optical flow field, by analyzing the motion of pixels between adjacent frames. The Lucas-Kanade optical flow method is a classic sparse optical flow estimation algorithm. It solves the motion speed of pixels by establishing the grayscale value relationship between pixels in adjacent frames and assuming that the motion of pixels in a local area is consistent.

[0042] The solution provided by the present invention combines the diamond search algorithm and the optical flow method. By applying the diamond search algorithm in the pre-processing of the optical flow method, real-time updating of the region of interest is achieved. The improved method effectively integrates the scale characteristics of the diamond search algorithm and the target-free characteristics of the optical flow method, and achieves accurate tracking of structural displacements. It effectively avoids the defect that the traditional optical flow method is limited to the assumption of small displacement measurement and cannot update the region of interest in real time to track large displacements.

[0043] The scheme of this embodiment is mainly aimed at the scenario where the target structure moves only in the two-dimensional plane where the image is located. That is, the target structure will only move up, down, left, and right in the plane where the image is located. This embodiment will collect the displacement video of the target structure, and perform frame processing based on the displacement video according to the time sequence in the video to obtain a sequence of video image frames to be processed. Since the target structure in this embodiment only moves up, down, left, and right in the plane where the image is located, the size of the target structure is the same in each video image frame of the displacement video, and only the displacement changes.

[0044] In this embodiment, the displacement video of the target structure is firstly processed by frame division to obtain a video image frame sequence to be processed, that is, the displacement video is decomposed into a group of independent image frames and converted into grayscale images to reduce the interference of factors such as color and lighting on the displacement tracking of the target structure.

[0045] Secondly, before performing the large diamond search and the small diamond search, the present embodiment can perform local scaling processing on the video image frame to obtain a local image including the target structure and the surrounding area as the updated video image frame to reduce the amount of diamond search calculation. Here, it is necessary to ensure that the same local scaling processing is performed on all video image frames in the video image frame sequence to ensure the accuracy of the final calculated displacement result.

[0046] This embodiment determines the search path for searching the video image frame based on the first search path of the large diamond search pattern and the second search path of the small diamond search pattern. The search path is determined in the following manner:

[0047] In one embodiment, the path determined based on the large diamond search mode and the small diamond search mode is: obtaining a first search path of the large diamond search mode, taking the origin coordinates of the position determined by the final displacement of the previous video image frame as the search starting point, and performing a large diamond search according to the first search path; for each search point on the first search path, comparing the two-dimensional correlation coefficient calculated for each search point with the two-dimensional correlation coefficient calculated for the previous search point, if the two-dimensional correlation coefficient calculated for the current search point is greater than the two-dimensional correlation coefficient calculated for the previous search point, then not performing a small diamond search for the previous search point, and continuing to perform a large diamond search for the next search point according to the first search path; if the two-dimensional correlation coefficient calculated for the current search point is not greater than the two-dimensional correlation coefficient calculated for the previous search point, then obtaining a second search path of the small diamond search mode, taking the previous search point as the search starting point, and performing a small diamond search according to the second search path; the obtaining of the two-dimensional correlation coefficients of all search path points includes: obtaining the two-dimensional correlation coefficients calculated for all search points in the large diamond search mode and the two-dimensional correlation coefficients calculated for all search points in the small diamond search mode.

[0048] like Figure 2 As shown, the large diamond search pattern has 9 search points in total, and the first search path formed by them is specifically:

[0049] The path formed by each point in the following calculation formula in sequence is the first search path of the large diamond search pattern:

[0050] Calculation formula (1).

[0051] Correspondingly, if Figure 3 As shown, the small diamond search pattern has a total of 5 search points, and the second search path formed by them is specifically:

[0052] The path formed by each point in the following calculation formula in sequence is the second search path of the small diamond search pattern:

[0053] Calculation formula (2).

[0054] Here, this embodiment adopts the order from right to left and from bottom to top to search the search points in the large diamond search pattern and the small diamond search pattern. When searching at each search point, the two-dimensional correlation coefficient between the calculation area corresponding to the search point and the area of ​​interest in the first video image frame is calculated.

[0055] In practical application, the two-dimensional correlation coefficient between the calculation area and the area of ​​interest in the first video image frame can be calculated specifically by the following calculation formula:

[0056] Calculation formula (3)

[0057] in, is the two-dimensional correlation coefficient, A is the calculation area, B is the region of interest in the first video image frame, m and n are the horizontal and vertical pixel dimensions of the image, respectively, is the mean pixel intensity of the calculation area, is the mean pixel intensity of the region of interest in the first video image frame.

[0058] The calculation formula for the two-dimensional correlation coefficient provided in this embodiment is highly robust to different noise and illumination conditions and can provide pixel-level measurement accuracy.

[0059] After calculating the two-dimensional correlation coefficients for all search points on the search path, the first calculation area corresponding to the maximum value in the two-dimensional correlation coefficient can be determined; according to the position coordinates of the first calculation area and the position coordinates of the region of interest in the first video image frame, the initial displacement of the target structure in each video image frame is determined. The initial displacement here is only a rough calculation of the displacement of the target structure, that is, the displacement direction and displacement distance of the target structure are preliminarily determined. Based on the preliminary displacement direction and displacement distance, the displacement of the target structure is further determined using the improved sparse optical flow method.

[0060] Unlike the diamond search method of template matching, the optical flow method provides a target-free method. Optical flow refers to the instantaneous speed of the movement of object pixels in a time-varying image. Since the optical flow contains the motion information of the object in the image, the change of pixels in the time domain and the correlation between adjacent frames can be used to find the correspondence between the previous frame and the current frame. There are two types of optical flow methods: sparse optical flow and dense optical flow. Since the target tracked in this embodiment is a partial area in the frame sequence of the target structure video image, it is only necessary to focus the tracking target on a specific point. Therefore, a sparse optical flow estimation algorithm, namely the Lucas-Kanade optical flow method, is adopted and improved to calculate the displacement of the target structure. Here, please refer to Figure 4 , Figure 4 Schematic diagram of optical flow calculation according to an embodiment of the present invention.

[0061] In the video image frames of this embodiment, since the first calculation region and the region of interest in each video image frame have the same area size, the first calculation region in each video image frame and the region of interest in the first video image frame contain the same number of pixels.

[0062] In this embodiment, the gray value of a pixel at a time point can be regarded as a ternary equation , where table indicates the horizontal coordinate of the pixel, represents the vertical coordinate of the pixel, Indicates time. The gray value of the pixel at the time point The next frame after that can be described as , where and Represent the small displacement in the horizontal and vertical directions respectively, represents the time increment. According to the principle of constant brightness, we can get formula (4), that is, the gray value of the pixel before and after the movement is constant. Expanding the right side of the above formula with Taylor formula, we can get formula (5), where represents the higher-order Peano remainder, which can be ignored. Substituting equation (5) into equation (4) can be simplified to equation (6).

[0063] Calculation formula (4)

[0064] Calculation formula (5)

[0065] Calculation formula (6)

[0066] In the above formulas (4), (5), and (6), , , Respectively represent the optical flow gradient of the image in the horizontal, vertical and time directions, is the time difference between the previous and next frames, that is, the index difference between adjacent frames is 1. represents the horizontal optical flow gradient, represents the vertical optical flow gradient, represents the optical flow gradient of the previous and next frames, then equation (6) can be written as:

[0067] Calculation formula (7)

[0068] According to the spatial consistency principle of the Lucas-Kanade optical flow method, that is, the pixels in the local area between adjacent frames maintain the same optical flow direction and the motion is consistent, it is assumed here that the image has n pixels in the local area, and the equations are combined to expand the calculation formula (7) into:

[0069] Calculation formula (8)

[0070] For the above optical flow gradient equations, this embodiment uses the following preset operator to calculate the horizontal optical flow gradient: and vertical optical flow gradient :

[0071] Calculation formula (9)

[0072] Calculation formula (10)

[0073] in, Representative i frame, , Represents the first pixel matrix of the calculation area j Row, No. k List, Represents the image sequence i The calculation area in the frame j Row, No. k The pixel grayscale value of the column.

[0074] Here, see Figure 5 , Figure 5 The figure is a schematic diagram of the average error analysis of different operators in the displacement measurement process for different sub-pixel scales. It can be seen from the figure that in the sub-pixel displacement calculation at different scales, the Barron operator provided in this embodiment has higher calculation accuracy and better robustness than the Horn operator, natural spline operator, and isotropic operator.

[0075] In this embodiment, the horizontal optical flow gradient is calculated and vertical optical flow gradient Substituting into the above optical flow gradient equations, that is, substituting the calculation structure of equations (9) and (10) into equation (8), and using the least squares method and matrix inverse calculation, the sub-pixel displacement of the target structure in the horizontal and vertical directions in each video image frame can be obtained. , That is, the fine displacement of the target structure in the horizontal and vertical directions in each video image frame.

[0076] By adding the initial displacement and the refined displacement, the final displacement of the target structure in each video image frame can be obtained.

[0077] The structural vibration displacement identification method based on diamond search and improved optical flow method proposed in this embodiment effectively overcomes the limitations of traditional optical flow method when the target motion is large by combining the diamond search algorithm with the Lucas-Kanade optical flow method, realizes accurate displacement tracking, and provides key support for structural displacement monitoring technology.

[0078] This embodiment uses the diamond search algorithm for integer pixel positioning, and combines it with the Lucas-Kanade optical flow method for sub-pixel displacement recognition to achieve high-precision displacement recognition. This embodiment uses the integer pixel matching results to update the region of interest of each frame in real time to ensure that the optical flow calculation area is synchronized with the target movement.

[0079] Below, the technical effect of this solution will be explained based on a specific application scenario.

[0080] This embodiment takes a pedestrian overpass in Changsha as an example, and applies the displacement identification method proposed in this embodiment to measure the displacement of the cable. A drone (DJIMavic 3) with a pixel resolution of 3840×2160 pixel and a frame rate of 60 fp is used to collect vibration video of the cable. The collection time is 1 minute, and the field test scene is shown in Figure 6 (a) and Figure 6 (b). Among them, Figure 6 (a) is a schematic diagram of the field test scene of the embodiment of the present invention facing the cable; Figure 6 (b) is a schematic diagram of the side instrument perspective of the field test scene of the embodiment of the present invention. This technology is used to identify the vibration displacement of the cable and the laser point through the drone video respectively, and the drone's own motion can be eliminated by subtracting the two to obtain the absolute displacement of the cable, such as Figure 7 shown.

[0081] The displacement results obtained by the fixed camera are used as reference values, and the cable displacement extracted by the drone is compared with the reference value, e.g. Figure 8 The results show that the displacement identified by the proposed algorithm and the displacement measured by the fixed camera have a fitting accuracy of 98.67%, which indicates that the displacement identification algorithm proposed in this embodiment has high accuracy.

[0082] In order to further verify the accuracy of the cable frequency obtained by this technology, the cable frequency obtained by this technology is compared with the actual frequency measured by the accelerometer, such as Fig. 9 , Fig.10 , as shown in Table 1. The results show that the cable frequency identification error is within 4%, which further verifies the accuracy of this method.

[0083] Table 1

[0084]

[0085] In view of the limitations of the traditional optical flow method in the displacement tracking range, this embodiment proposes an accurate displacement tracking method integrating a diamond search algorithm. By performing integer pixel matching on the target areas of adjacent frames, the region of interest of each frame is updated and located in real time. This solves the deviation in the optical flow gradient calculation results caused by excessive target motion and realizes sub-pixel tracking of vibrations under different motion modes.

[0086] The structural vibration displacement identification method based on diamond search and improved optical flow method provided by the embodiment of the present invention obtains a video image frame sequence to be processed; the video image frame sequence is obtained by performing frame processing on the displacement video of the target structure in time sequence; the region of interest of the first video image frame in the video image frame sequence is determined, the first video image frame is the first video image frame in the video image frame sequence, and the region of interest is the region where the target structure is located in the first video image frame; based on the region of interest, the pixel coordinates of the upper left corner point of the region of interest are determined as the origin coordinates of the region of interest; for each of the remaining video image frames except the first video image frame in the video image frame sequence, the origin coordinates of the final displacement determined position of the previous video image frame are used as the search starting point, the path determined by the large diamond search mode and the small diamond search mode is used as the search path, and the area size of the region of interest is used as the coordinate of each path point The calculation area is used to perform large diamond search and small diamond search, and the two-dimensional correlation coefficient between the calculation area and the region of interest in the first video image frame is calculated at each search path point; the two-dimensional correlation coefficients of all search path points are obtained, and the first calculation area corresponding to the maximum value of the two-dimensional correlation coefficients is determined; according to the position coordinates of the first calculation area and the position coordinates of the region of interest in the first video image frame, the initial displacement of the target structure in each video image frame is determined; for each video image frame except the first video image frame in the video image frame sequence, an optical flow gradient equation group is constructed based on the first calculation area in each video image frame and the region of interest in the first video image frame, and the optical flow gradient equation group is calculated using a preset operator to determine the fine displacement of the target structure in each video image frame; the initial displacement and the fine displacement are added to determine the final displacement of the target structure in each video image frame. The scheme provided by the present invention effectively integrates the scale characteristics of the diamond search algorithm and the target-free characteristics of the optical flow method, realizes the accurate tracking of the structural displacement, and effectively avoids the defect that the traditional optical flow method is limited to the assumption of small displacement measurement and cannot update the region of interest in real time to track large displacement.

[0087] This embodiment combines the diamond search algorithm and the optical flow method to achieve accurate tracking of target displacement, and can track both small and large displacements, effectively overcoming the limitations of the traditional optical flow method when the target moves more. The two-dimensional correlation coefficient calculation method used in this embodiment has strong robustness to different noise and lighting conditions, and can adapt to complex and changing environments. This embodiment can be used in multiple fields such as structural displacement monitoring and target tracking, and has high measurement accuracy and certain versatility.

[0088] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0089] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A structural vibration displacement identification method based on diamond search and improved optical flow method, characterized in that: The method comprises: Acquire a video image frame sequence to be processed; the video image frame sequence is obtained by performing frame processing on the displacement video of the target structure in time sequence; Determine a region of interest of a first video image frame in the video image frame sequence, wherein the first video image frame is the first video image frame in the video image frame sequence, and the region of interest is a region where a target structure is located in the first video image frame; based on the region of interest, determine the pixel coordinates of the upper left corner point of the region of interest as the origin coordinates of the region of interest; For each of the remaining video image frames except the first video image frame in the video image frame sequence, the coordinates of the origin of the final displacement position of the previous video image frame are used as the search starting point, the path determined by the large diamond search mode and the small diamond search mode is used as the search path, and the area size of the region of interest is used as the calculation area of ​​each path point, and a large diamond search and a small diamond search are performed to calculate the two-dimensional correlation coefficient between the calculation area and the region of interest in the first video image frame at each search path point; the two-dimensional correlation coefficients of all search path points are obtained, and the first calculation area corresponding to the maximum value of the two-dimensional correlation coefficients is determined; according to the position coordinates of the first calculation area and the position coordinates of the region of interest in the first video image frame, the initial displacement of the target structure in each video image frame is determined; For each remaining video image frame in the video image frame sequence except the first video image frame, construct an optical flow gradient equation group based on a first calculation region in each video image frame and a region of interest in the first video image frame, and calculate the optical flow gradient equation group using a preset operator to determine a fine displacement of a target structure in each video image frame; Adding the initial displacement and the refined displacement to determine a final displacement of the target structure in each video image frame; Among them, the path determined based on the large diamond search mode and the small diamond search mode is: obtaining a first search path of the large diamond search mode, taking the origin coordinates of the position determined by the final displacement of the previous video image frame as the search starting point, and performing a large diamond search according to the first search path; for each search point on the first search path, comparing the two-dimensional correlation coefficient calculated for each search point with the two-dimensional correlation coefficient calculated for the previous search point, if the two-dimensional correlation coefficient calculated for the current search point is greater than the two-dimensional correlation coefficient calculated for the previous search point, then not performing a small diamond search for the previous search point, and continuing to perform a large diamond search for the next search point according to the first search path; if the two-dimensional correlation coefficient calculated for the current search point is not greater than the two-dimensional correlation coefficient calculated for the previous search point, then obtaining a second search path of the small diamond search mode, taking the previous search point as the search starting point, and performing a small diamond search according to the second search path; the obtaining of the two-dimensional correlation coefficients of all search path points includes: obtaining the two-dimensional correlation coefficients calculated for all search points in the large diamond search mode and the two-dimensional correlation coefficients calculated for all search points in the small diamond search mode.

2. The structural vibration displacement identification method based on diamond search and improved optical flow method according to claim 1 is characterized in that: The first search path of the large diamond search pattern is: The path formed by each point in the following calculation formula in sequence is the first search path of the large diamond search pattern: Calculation formula (1).

3. The structural vibration displacement identification method based on diamond search and improved optical flow method according to claim 2 is characterized in that: The second search path of the small diamond search pattern is: The path formed by each point in the following calculation formula in sequence is the second search path of the small diamond search pattern: Calculation formula (2).

4. The structural vibration displacement identification method based on diamond search and improved optical flow method according to claim 1 is characterized in that: The calculating, at each search path point, of a two-dimensional correlation coefficient between the calculation area and the area of ​​interest in the first video image frame comprises: The two-dimensional correlation coefficient between the calculation area and the region of interest in the first video image frame is calculated using the following calculation formula: Calculation formula (3) in, is the two-dimensional correlation coefficient, A is the calculation area, B is the region of interest in the first video image frame, m and n are the horizontal and vertical pixel dimensions of the image, respectively, is the mean pixel intensity of the calculation area, is the mean pixel intensity of the region of interest in the first video image frame.

5. The structural vibration displacement identification method based on diamond search and improved optical flow method according to claim 1 is characterized in that: Constructing an optical flow gradient equation group based on the first calculation area in each video image frame and the region of interest in the first video image frame, including: The constructed optical flow gradient equations are shown in the following formula: Calculation formula (4) in, is the gray value of a pixel among the n pixels in the region of interest, represents the horizontal coordinate of the pixel, represents the vertical coordinate of the pixel, Indicates time, is the gray value of the corresponding pixel in the first calculation area, and Respectively represent the small displacement of the first calculation area relative to the region of interest in the horizontal and vertical directions, represents the time increment, , Represents the higher level of Peano, , , , Respectively represent the optical flow gradient of the image in the horizontal, vertical and time directions, and use , , express , , , , is the optical flow gradient of each pixel.

6. The structural vibration displacement identification method based on diamond search and improved optical flow method according to claim 1 is characterized in that: The step of calculating the optical flow gradient equations using a preset operator to determine the fine displacement of the target structure in each video image frame includes: Use the following preset operator to calculate the horizontal optical flow gradient and vertical optical flow gradient : Calculation formula (5) Calculation formula (6) in, Representative i frame, , Represents the first pixel matrix of the calculation area j Row, No. k List, Represents the image sequence i The calculation area in the frame j Row, No. k The pixel grayscale value of the column; The horizontal optical flow gradient calculated and vertical optical flow gradient Substitute into the optical flow gradient equation group, and use the least square method and matrix inverse operation to calculate and obtain the fine displacement of the target structure in the horizontal and vertical directions in each video image frame.

Images