Large-scale structure three-dimensional full-field deformation measurement image processing method and system

Abstract

The application discloses a large-scale structure three-dimensional full-field deformation measurement image processing method and system, relates to the technical field of image processing, and has the following points: a binocular image sequence is constructed, and a polar line function relationship of each measurement point in a left camera image in a corresponding right camera image is acquired; a matching point corresponding to each measurement point is searched in a one-dimensional search region; based on a three-dimensional state vector of each measurement point at a previous frame moment, a Kalman filtering method is used to predict a state vector of each measurement point at a current frame moment, and a corresponding three-dimensional prediction state vector at the current frame moment is obtained; a deviation between the three-dimensional prediction state vector at the current frame moment and a three-dimensional state vector is calculated by using an association cost function, and the three-dimensional state vector at the current frame moment is updated based on a difference value; after the three-dimensional state vectors of all measurement points at different frame moments are updated, the three-dimensional state vectors at all frame moments are used to generate a three-dimensional motion trajectory of the measurement points.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image processing method and system for measuring the three-dimensional full-field deformation of large-scale structures. Background Technology

[0002] Large-scale engineering structures, such as high-voltage transmission towers, long-span bridges, towers, cables, and beams, are subject to long-term effects from wind loads, vehicle loads, temperature changes, and foundation disturbances during their service life, which can easily lead to dynamic displacement, vibration, deflection, and local deformation. These deformation parameters are important bases for structural safety assessment, service status monitoring, and life prediction.

[0003] Existing methods for monitoring structural deformation mainly include contact measurement methods and non-contact optical measurement methods. Contact measurement methods typically use sensors such as displacement gauges, deflectometers, accelerometers, and strain gauges. While these methods are technologically mature, they have significant shortcomings in large-scale structural field applications: First, sensor installation and wiring are difficult, especially on bridges hundreds of meters high, tall power towers, or structures spanning the sky, resulting in high installation and maintenance costs and significant safety risks. Second, sensors can only acquire information from a limited number of measurement points, making it difficult to reflect the overall deformation of the structural surface. Third, contact sensors and their fixing devices may impose additional loads on lightweight or flexible structures, affecting the accuracy of the measurements.

[0004] With the development of machine vision and image processing technologies, non-contact measurement technologies such as video displacement gauges and digital image correlation methods are increasingly being used for structural deformation measurement. Video displacement gauges, in particular, can measure parameters such as displacement, angle, and deflection using high-resolution industrial cameras and real-time image processing algorithms. They offer advantages such as being non-contact, adaptable to outdoor environments, utilizing natural texture features, and capable of sub-millimeter-level measurements over distances of up to 100 meters. However, traditional visual measurement methods still face the following challenges in large-scale engineering applications:

[0005] First, traditional two-dimensional video displacement measurements are mostly targeted at single points or a few key points, making it difficult to obtain three-dimensional full-field deformation information of the structural surface. For spatial structures such as bridges and transmission towers, measuring only two-dimensional planar displacement cannot accurately reflect their spatial deformation characteristics;

[0006] Second, traditional digital image correlation methods involve a large amount of computation when performing a global two-dimensional search in large-format images. For images with tens of millions of pixels and high frame rate dynamic monitoring scenarios, it is difficult to meet the real-time computing requirements if a two-dimensional global search is performed directly on each correlation sub-region.

[0007] Third, large-scale outdoor measurements take place in complex environments, with monitoring distances potentially reaching hundreds of meters. On-site challenges include strong light, backlighting, fog, wind-induced vibrations, complex backgrounds, and partial obstructions. In field tests on long-span bridges, common conditions include fog, vehicle vibrations, and natural wind; the test field of view can reach tens of meters, and the working distance can reach hundreds of meters. Transmission tower sites also present typical outdoor conditions such as strong light, natural wind vibrations, and large-format measurements.

[0008] Fourth, traditional algorithms based on grayscale correlation or local feature matching are prone to problems such as mismatch, feature loss, and trajectory jump when the target is rotated, stretched, sheared, scaled, or subject to changes in local illumination or short-term occlusion, resulting in insufficient continuity and reliability of long-term dynamic monitoring.

[0009] Fifth, existing methods typically perform trajectory association in a two-dimensional image plane, failing to fully utilize the three-dimensional object space information obtained from binocular reconstruction. When the structure undergoes spatial motion (especially displacement in the depth direction), the proximity relationships of feature points in the two-dimensional image plane will change accordingly, leading to data association failure and decreased stability of multi-target tracking.

[0010] Based on this, the present invention aims to provide a method and system for image processing of large-scale structural three-dimensional full-field deformation measurement, so as to solve the related problems mentioned above. Summary of the Invention

[0011] The present invention aims to provide an image processing method and system for large-scale structural three-dimensional full-field deformation measurement. It utilizes binocular camera calibration parameters to establish epipolar constraint relationships between left and right images, transforming the two-dimensional global search in traditional digital image correlation methods into a one-dimensional strip search along the epipolar direction, thus significantly reducing the search range and computational load. Simultaneously, a box-filtered window incremental update method is introduced during correlation coefficient calculation. By updating only the pixel rows or columns entering and leaving the search window, it avoids repeated summation of the entire correlation sub-region, improving integer pixel matching efficiency. Furthermore, sub-pixel positioning and adaptive geometric correction of the correlation sub-region improve matching accuracy when the structural surface undergoes rotation, stretching, shearing, or perspective changes. The three-dimensional object coordinates obtained from binocular reconstruction are used as multi-target tracking state variables. Combined with Kalman prediction and data association methods, continuous tracking of the measurement point trajectory is achieved, and the predicted position is fed back to the correlation search area of ​​the next frame image. This transforms correlation matching from a passive global search to a prediction-guided local search, thereby reducing measurement interruptions caused by mismatches, trajectory jumps, and short-term occlusion, and improving the robustness and real-time performance of large-scale structural three-dimensional displacement field measurement.

[0012] This invention is achieved through the following technical solution:

[0013] A method for image processing of large-scale structural three-dimensional full-field deformation measurement, the method comprising:

[0014] Construct a stereo image sequence; wherein, the first frame of the stereo image sequence is a reference stereo image of the structure under test in a reference state, the second frame of the stereo image and the subsequent multiple frames of the stereo image are multiple sets of dynamic stereo images of the structure under test in a load state, and the stereo image includes a left camera image and a right camera image;

[0015] Multiple measurement points are selected in the left camera image of each frame of stereo image, and the epipolar function relationship of each measurement point in the left camera image in the corresponding right camera image is obtained to construct a one-dimensional search region in the right camera image; and the integer pixel fast correlation search method of box filtering is used to search for the matching point corresponding to each measurement point in the one-dimensional search region.

[0016] By using the bi-target positioning parameters and projection matrix, combined with the position parameters of the measurement points and the position parameters of the corresponding matching points, the three-dimensional state vector of each measurement point at each frame time is obtained.

[0017] Based on the three-dimensional state vector of each measurement point in the previous frame, the Kalman filter method is used to predict the state vector of each measurement point in the current frame, so as to obtain the three-dimensional predicted state vector corresponding to the current frame.

[0018] The difference between the 3D predicted state vector and the 3D state vector at the current frame time is calculated using the correlation cost function, and the 3D state vector at the current frame time is updated based on the difference value.

[0019] After updating the three-dimensional state vectors of all measurement points at different frame times, the three-dimensional motion trajectory of the measurement points is generated using the three-dimensional state vectors at all frame times; spatial fitting is performed on the three-dimensional motion trajectory of all measurement points to obtain the corrected image of the structure under test.

[0020] Furthermore, a binocular image sequence is constructed, specifically as follows:

[0021] A binocular vision measurement unit was constructed using two image acquisition devices, and the binocular vision measurement unit was calibrated.

[0022] A reference binocular image is obtained by acquiring reference left and reference right camera images of the structure under test in a reference state using a binocular vision measurement unit.

[0023] Using a binocular vision measurement unit, multiple dynamic left and right camera images of the structure under load are acquired at preset time intervals to obtain multiple sets of dynamic binocular images.

[0024] The reference stereo image is used as the first frame of the stereo image in the sequence. Multiple sets of dynamic stereo images are arranged after the first frame of the stereo image according to the acquisition time sequence to construct the stereo image sequence.

[0025] Furthermore, multiple measurement points are selected in the left camera image of each frame of the stereo image, specifically:

[0026] In each frame of the binocular image, select the region of interest of the structure under test, divide the region of interest according to the preset region division rules, and take the center point of each relevant sub-region as the measurement point.

[0027] Furthermore, a box-filtered integer-pixel fast correlation search method is employed to find the matching point corresponding to each measurement point within a one-dimensional search region, specifically as follows:

[0028] Each measurement point is located in a sub-region as a reference sub-region. The gray value of each reference sub-region is obtained, and the gray sum and gray square sum of the reference sub-region are calculated.

[0029] In the direction of the one-dimensional search area corresponding to each measurement point, the pre-constructed search window is moved according to the preset moving step size, and the area after each search window movement is taken as a candidate sub-region. The box filtering incremental update strategy is used to obtain the gray sum, gray square sum and gray product sum between the candidate sub-region and the corresponding reference sub-region after each movement.

[0030] The sum of gray levels, the sum of squared gray levels, and the sum of gray-level products of each candidate sub-region are compared with the sum of gray levels and the sum of squared gray levels of the corresponding reference sub-region. These are then input into the cross-correlation coefficient calculation function to calculate the correlation coefficient between each candidate sub-region and the reference sub-region.

[0031] The center point of the candidate sub-region with the highest correlation coefficient is selected as the matching point of the corresponding measurement point.

[0032] Furthermore, after obtaining the corresponding matching point at each measurement point, the method also includes:

[0033] Based on the peak correlation coefficient and neighborhood correlation value of the matching point, the position of the matching point is corrected by a quadratic polynomial curve fitting method to obtain the corrected position of the matching point.

[0034] Furthermore, the three-dimensional state vector at the current frame time is updated based on the difference value, specifically as follows:

[0035] When the difference value is not less than the preset threshold, the three-dimensional predicted state vector is replaced with the three-dimensional state vector at the current frame time.

[0036] When the difference value is less than the preset threshold, the three-dimensional state vector will be retained at the current frame time.

[0037] The present invention also provides a large-scale structure three-dimensional full-field deformation measurement image processing system, which is used in the large-scale structure three-dimensional full-field deformation measurement image processing method described in any one of the above claims, the system comprising:

[0038] An image sequence construction module is used to construct a stereo image sequence; wherein, the first frame of the stereo image sequence is a reference stereo image of the structure under test in a reference state, and the second frame of the stereo image and the subsequent multiple frames of the stereo image are multiple sets of dynamic stereo images of the structure under test in a load state, and the stereo image includes a left camera image and a right camera image.

[0039] The matching point search module is used to select multiple measurement points in the left camera image of each frame of stereo image and obtain the epipolar function relationship of each measurement point in the left camera image in the corresponding right camera image to construct a one-dimensional search region in the right camera image; and adopts the box filtering integer pixel fast correlation search method to search for the matching point corresponding to each measurement point in the one-dimensional search region.

[0040] The three-dimensional vector calculation module is used to measure the three-dimensional state vector of each measurement point at each frame time by using the bi-target positioning parameters and projection matrix, combined with the position parameters of the measurement points and the position parameters of the corresponding matching points.

[0041] The 3D vector prediction module is used to predict the state vector of each measurement point in the current frame based on the 3D state vector of each measurement point in the previous frame, using the Kalman filter method, so as to obtain the 3D predicted state vector corresponding to the current frame.

[0042] The 3D vector update module is used to calculate the difference between the 3D predicted state vector and the 3D state vector at the current frame time using the correlation cost function, and update the 3D state vector at the current frame time based on the difference value.

[0043] The image fitting processing module is used to update the three-dimensional state vectors of all measurement points at different frame times, and then use the three-dimensional state vectors at all frame times to generate the three-dimensional motion trajectory of the measurement points; and to perform spatial fitting on the three-dimensional motion trajectory of all measurement points to obtain the corrected image of the structure under test.

[0044] The present invention also provides a computer device, including a system memory and a processor, wherein the system memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0045] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0046] The present invention also provides a computer program product containing instructions that, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method described in any of the preceding claims.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] In this invention, epipolar constraint relationships between left and right images are established using binocular camera calibration parameters, transforming the two-dimensional global search in traditional digital image correlation methods into a one-dimensional strip search along the epipolar direction, thereby significantly reducing the search range and computational load. Simultaneously, a box-filtered window incremental update method is introduced during correlation coefficient calculation, updating only the pixel rows or columns entering and leaving the search window to avoid repeated summation of the entire correlation sub-region, thus improving integer-pixel matching efficiency. Furthermore, sub-pixel localization and adaptive geometric correction of the correlation sub-region improve matching accuracy when structural surfaces undergo rotation, stretching, shearing, or perspective changes. The three-dimensional object coordinates obtained from binocular reconstruction are used as multi-target tracking state variables, combined with Kalman prediction and data association methods to achieve continuous tracking of measurement point trajectories. The predicted position is fed back to the correlation search area of ​​the next frame image, transforming correlation matching from a passive global search to a prediction-guided local search. This reduces measurement interruptions caused by mismatches, trajectory jumps, and short-term occlusion, improving the robustness and real-time performance of large-scale structural three-dimensional displacement field measurements. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0050] Figure 1 This is a flowchart illustrating an image processing method for measuring the three-dimensional full-field deformation of a large-scale structure in this embodiment.

[0051] Figure 2 This is a schematic diagram illustrating the principle of binocular vision 3D reconstruction in this embodiment;

[0052] Figure 3 This is a schematic diagram of window movement for the box filtering incremental update strategy in this embodiment;

[0053] Figure 4 This is a schematic diagram of the one-dimensional search area in this embodiment;

[0054] Figure 5This is a schematic diagram of the module connections of a large-scale structural three-dimensional full-field deformation measurement image processing system in this embodiment;

[0055] Figure 6 This is a schematic diagram of the structure of a computer device in this embodiment. Detailed Implementation

[0056] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0057] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0058] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0059] Example 1

[0060] join Figure 1 , Figure 1 A flowchart illustrating an image processing method for large-scale structural three-dimensional full-field deformation measurement is shown, wherein the method includes:

[0061] S1: Construct a stereo image sequence; wherein, the first frame of the stereo image sequence is a reference stereo image of the structure under test in a reference state, the second frame of the stereo image and the subsequent multiple frames of the stereo image are multiple sets of dynamic stereo images of the structure under test in a load state, and the stereo image includes a left camera image and a right camera image.

[0062] Specifically, in this embodiment, a binocular image sequence is constructed as follows:

[0063] First, a binocular vision measurement unit is constructed using two image acquisition devices, and then the binocular vision measurement unit is calibrated.

[0064] Next, the binocular vision measurement unit is used to acquire the reference left camera image and the reference right camera image of the structure under test in the reference state to obtain the reference binocular image;

[0065] Then, using a binocular vision measurement unit, multiple dynamic left camera images and dynamic right camera images of the structure under load are acquired at preset time intervals to obtain multiple sets of dynamic binocular images.

[0066] Finally, the reference stereo image is used as the first frame of the stereo image in the sequence, and multiple sets of dynamic stereo images are arranged after the first frame of the stereo image according to the acquisition time sequence to construct the stereo image sequence.

[0067] It should be noted that in this embodiment, the image acquisition device uses a high-resolution area array industrial camera. The camera resolution, lens focal length, baseline distance, and acquisition frequency can be determined based on the test distance, field of view, structural dimensions, and desired measurement accuracy. Two industrial cameras are then positioned at a certain distance in front of the large-scale structure being measured. The binocular vision measurement unit is calibrated using a calibration board, calibration cross, or on-site feature calibration method. Simultaneously, the intrinsic, extrinsic, and distortion parameters of the binocular vision measurement unit are calibrated to obtain the focal length, principal point, distortion coefficients, rotation matrix, translation vector, and projection matrix of the binocular system for both cameras. After calibration, a unified three-dimensional object coordinate system is established, enabling any feature point on the structural surface to be reconstructed into three-dimensional spatial coordinates from the measurement points and matching points in the left and right camera images. See [link to relevant documentation]. Figure 2 , Figure 2 A schematic diagram illustrating the principle of binocular vision 3D reconstruction is shown.

[0068] It should also be noted that in this embodiment, the reference state refers to the stable state of the structure under test without any load, while the load state refers to the dynamic state of the structure under test under wind load, vehicle load, or other dynamic load. Under the reference state, the binocular vision measurement unit acquires only one binocular image of the structure under test, which serves as the reference reference image for subsequent dynamic binocular images. Under the load state, the binocular vision unit will continuously acquire multiple sets of binocular images of the structure under test over a certain period of time, which will serve as the objects for subsequent dynamic correction processing.

[0069] Meanwhile, in a stereo image sequence, since dynamic stereo images are acquired in chronological order, each frame can also be called a frame moment; therefore, the first frame is the first frame moment, the second frame is the second frame moment, and so on.

[0070] S2: Select multiple measurement points in the left camera image of each frame of stereo image, and obtain the epipolar function relationship of each measurement point in the left camera image in the corresponding right camera image to construct a one-dimensional search region in the right camera image; and use the box filtering integer pixel fast correlation search method to search for the matching point corresponding to each measurement point in the one-dimensional search region.

[0071] Specifically, in this embodiment, multiple measurement points are selected in the left camera image of each frame of binocular images, as follows:

[0072] First, select the region of interest (ROI) of the structure under test in the left camera image of each frame of binocular image, and divide the ROI according to the preset region division rules, and take the center point of each relevant sub-region as the measurement point.

[0073] Then, the epipolar function relationship of each measurement point in the left camera image is obtained in the corresponding right camera image to construct a one-dimensional search region in the right camera image;

[0074] Finally, a box-filtering-based integer-pixel fast correlation search method is used to find the matching point corresponding to each measurement point within a one-dimensional search area.

[0075] It should be noted that in this embodiment, the preset region division rule includes dividing regions according to a preset division interval or dividing regions according to a random division method; in other embodiments, other region division methods may also be used.

[0076] It should also be noted that, in this embodiment, for the measurement point in the left camera image at any frame time in the binocular image sequence, the corresponding epipolar function relationship in the right camera image at the same time is calculated based on the binocular calibration parameters; thus, the matching search in the right camera image no longer performs a two-dimensional global search, but rather a search is performed in a one-dimensional banded region near the corresponding epipolar line; at the same time, the binocular calibration parameters are obtained during the calibration process of the binocular vision measurement unit.

[0077] Specifically, in this embodiment, for any frame of the binocular image sequence, firstly, a region of interest (ROI) of the structure to be measured is selected in the left camera image of each frame of the binocular image. During the region selection process, background, occlusions, and non-measurement areas are removed. Then, the ROI is divided into multiple related sub-regions according to a preset sub-region interval or a random division method, and the center point of the related sub-region is used as the measurement point. Then, the epipolar line of each measurement point is obtained in the corresponding right camera image, and the epipolar line is used as a one-dimensional search region for candidate matching points. That is, the candidate matching points in the right camera image and the corresponding measurement points conform to the following epipolar function relationship: In the formula, Indicates candidate matching points, , Indicates the measurement point. ; Candidate matching points in the right camera image should satisfy the following conditions: ;

[0078] Meanwhile, in this embodiment, a box-filtered integer-pixel fast correlation search method is used to search for the matching point corresponding to each measurement point within a one-dimensional search area, as follows:

[0079] Each measurement point is located in a sub-region as a reference sub-region. The gray value of each reference sub-region is obtained, and the gray sum and gray square sum of the reference sub-region are calculated.

[0080] In the direction of the one-dimensional search area corresponding to each measurement point, the pre-constructed search window is moved according to the preset moving step size, and the area after each search window movement is taken as a candidate sub-region. The box filtering incremental update strategy is used to obtain the gray sum, gray square sum and gray product sum between the candidate sub-region and the corresponding reference sub-region after each movement.

[0081] The sum of gray levels, the sum of squared gray levels, and the sum of gray-level products of each candidate sub-region are compared with the sum of gray levels and the sum of squared gray levels of the corresponding reference sub-region. These are then input into the cross-correlation coefficient calculation function to calculate the correlation coefficient between each candidate sub-region and the reference sub-region.

[0082] The center point of the candidate sub-region with the highest correlation coefficient is selected as the matching point of the corresponding measurement point.

[0083] It should be noted that, in this embodiment, see Figure 3 , Figure 3 The diagram illustrates the window movement of the box filtering incremental update strategy. The box filtering incremental update strategy involves adding new pixel columns or rows entering the search window and subtracting pixel columns or rows leaving the search window during each movement. This allows for rapid updates of the grayscale sum, grayscale square sum, and grayscale product sum within the search window, thereby avoiding repeated traversal of the entire search window.

[0084] It should also be noted that in this embodiment, the preset movement step size is set to one pixel column. In other embodiments, other step size settings can also be used. At the same time, the search window adopts a 6×6 rectangular search window. In other embodiments, other window sizes can also be used.

[0085] Specifically, in this embodiment, see Figure 4 , Figure 4 A schematic diagram of a one-dimensional search region is shown. In the stereo image at any frame time, taking a measurement point in the left camera image as an example, the sub-region where the measurement point in the left camera image is located is taken as the reference sub-region, and the gray value of the reference sub-region is obtained. The gray levels of the reference sub-region are calculated. and grayscale sum of squares Then, a search window is set on the one-dimensional search area in the right camera image to obtain the initial candidate sub-region; the gray value of the sub-region of the initial candidate sub-region is obtained. , Represent the candidate displacement and obtain the gray level of the initial candidate sub-region. Sum of gray levels and the gray-level product between the corresponding reference sub-region and the sum Then, move the pre-built search window once according to the preset step size to obtain the updated candidate sub-regions, and update the gray sum, gray square sum, and gray product sum of the candidate sub-regions. Specifically: , , In the formula, This represents the grayscale sum of the updated candidate sub-regions. This represents the grayscale of the pixel column that is removed from the search window. This represents the grayscale value of the pixel column that is moved into the search window. This represents the sum of squared gray levels of the updated candidate sub-regions. This represents the sum of the squared gray levels of the pixel columns that have been moved out of the search window. This represents the sum of the squared gray levels of the pixel column that is moved into the search window. This represents the sum of the grayscale products of the updated candidate sub-regions. This represents the sum of the grayscale products of the pixel columns that have been moved out of the search window. This represents the sum of grayscale products of the pixel columns moved into the search window; finally, the search window is moved sequentially over the one-dimensional search area in the same manner as described above, and the sum of grayscale, the sum of squared grayscale, and the sum of grayscale products of each subsequent candidate sub-region are calculated and updated.

[0086] Finally, by inputting the sum of gray levels, the sum of squared gray levels, and the sum of gray-level products of each candidate sub-region, along with the sum of gray levels and the sum of squared gray levels of the corresponding reference sub-region, into the cross-correlation coefficient calculation function, the correlation coefficient between each candidate sub-region and the reference sub-region is calculated. The cross-correlation coefficient calculation function is as follows: The center point of the candidate sub-region with the highest correlation coefficient is selected as the matching point of the corresponding measurement point.

[0087] In addition, in this embodiment, after obtaining the matching point corresponding to each measurement point, the matching point is also corrected, as follows: based on the peak value of the correlation coefficient and the neighborhood correlation value of the matching point, the position of the matching point is corrected by the quadratic polynomial curve fitting method to obtain the corrected matching point position.

[0088] The quadratic polynomial curve relation is: In the formula, This represents the local coordinates near the peak value of the matching point. , , , , , All represent the fitting coefficients; let the correlation surface The first-order partial derivative is zero, specifically: Find the extreme points of the surface Using the extreme points of the surface as the correction amount for the displacement of the matching point, the corrected displacement of the matching point is: In the formula, This indicates the corrected displacement. This represents the displacement before correction. The corrected displacement is used to correct the position of the matching point, resulting in the corrected position of the matching point.

[0089] S3: Using the dual-target positioning parameters and projection matrix, combined with the position parameters of the measurement points and the position parameters of the corresponding matching points, the three-dimensional state vector of each measurement point at each frame time is obtained.

[0090] It should be noted that, in this embodiment, for the matched measurement points and matching points in the left and right camera images, triangulation or least squares forward intersection is performed based on the binocular positioning parameters and projection matrix to obtain the three-dimensional coordinates and state vectors of each measurement point at each frame time of the binocular image sequence; the current three-dimensional coordinates corresponding to any frame to be processed are subtracted from the reference three-dimensional coordinates corresponding to the reference frame to obtain the three-dimensional displacement components of each measurement point relative to the reference state.

[0091] Specifically, in this embodiment, for the matched measurement points in the left and right camera images and matching points According to the projection matrix of the binocular vision measurement unit and Three-dimensional space point satisfy , In the formula, , The scale factor is represented; then the three-dimensional coordinates of the measurement point are obtained through triangulation or least-squares forward intersection. ; using the three-dimensional coordinates of the first frame Based on the baseline, the three-dimensional displacement at the current frame time is: ,in, These represent the displacements of the measuring point in three directions within the three-dimensional object-space coordinate system; simultaneously, the displacements of the first... Each measurement point at frame time The three-dimensional state vector is represented as: In the formula, For at frame time Time measurement point Spatial coordinates in a three-dimensional object-space coordinate system Let it be its velocity components in three directions.

[0092] S4: Based on the three-dimensional state vector of each measurement point at the time of the previous frame, the Kalman filter method is used to predict the state vector of each measurement point at the time of the current frame, so as to obtain the three-dimensional predicted state vector corresponding to the time of the current frame.

[0093] Specifically, in this embodiment, the Kalman filter method is used to predict the state vector of each measurement point at the current frame time, thereby obtaining the three-dimensional predicted state vector corresponding to the current frame time, as follows: In the formula, Indicates measurement point At the current frame time The three-dimensional predicted state vector, Indicates measurement point In the previous frame The three-dimensional state vector, Indicates process noise. Represents the state transition matrix. In the formula, This represents the time interval between two adjacent frames.

[0094] The candidate 3D measurement points obtained from the stereo reconstruction of the current frame are denoted as: Where j is the candidate measurement point number in the current frame. The measurement equation can be expressed as: ,in, To measure noise, For the observation matrix: .

[0095] S5: Calculate the difference between the 3D predicted state vector and the 3D state vector at the current frame time using the correlation cost function, and update the 3D state vector at the current frame time based on the difference value.

[0096] Specifically, in this embodiment, at each frame, the difference between the predicted 3D state vector and the measured 3D state vector is calculated. The correlation cost function used for the difference calculation is as follows: In the formula, Represents the three-dimensional predicted state vector With three-dimensional state vector The deviation between them , , , , All represent weighting coefficients. Represents the three-dimensional predicted state vector With three-dimensional state vector Spatial distance between them In the formula, This represents the spatial location of the three-dimensional predicted state vector. Represents the spatial location of a three-dimensional state vector; Represents the three-dimensional predicted state vector With three-dimensional state vector The difference in the direction of movement between them In the formula, Represents the three-dimensional predicted state vector The predicted velocity direction, Represents a three-dimensional state vector The direction of velocity, Represents the relevant matching coefficient. This indicates the binocular reconstruction error. Indicates the confidence level of historical trajectories;

[0097] When the calculated difference value is not less than the preset threshold, the three-dimensional predicted state vector is replaced with the three-dimensional state vector at the current frame time; when the difference value is less than the preset threshold, the three-dimensional state vector is retained at the current frame time; after determining the three-dimensional state vector at the current frame time, the state vector at the next frame time is predicted using the three-dimensional state vector at the current frame time.

[0098] S6: After updating the three-dimensional state vectors of all measurement points at different frame times, the three-dimensional motion trajectory of the measurement points is generated using the three-dimensional state vectors at all frame times; the three-dimensional motion trajectory of all measurement points is spatially fitted to obtain the corrected image of the structure under test.

[0099] Specifically, in this embodiment, after updating the three-dimensional state vectors of all measurement points at different frame times, the three-dimensional motion trajectory of the measurement points is generated using the three-dimensional state vectors at all frame times. In the formula, Indicates measurement point The system obtains the three-dimensional motion trajectory; it performs spatial fitting on the three-dimensional motion trajectories of multiple measurement points to generate various visualization forms such as three-dimensional full-field displacement cloud maps, deflection curves, displacement time history curves, and deformation animations of the structural surface. B-spline curve or B-spline surface fitting methods are used to smooth the noisy displacement field, and weights are set according to parameters such as correlation coefficient, tracking confidence, and reconstruction error to reduce the impact of low-quality measurement points on the overall results.

[0100] It should be noted that, in this embodiment, a trajectory confidence level is also set for the three-dimensional motion trajectory corresponding to each measurement point. This confidence level can be determined comprehensively based on the relevant matching coefficients, 3D reconstruction error, number of consecutive tracking frames, and historical matching stability, specifically as follows: In the formula, The correlation matching coefficients for the current frame of this trajectory. For 3D reconstruction error, For continuous tracking time or number of continuous tracking frames, As an indicator of the smoothness of historical trajectories, These are the weighting coefficients.

[0101] Specifically, the three-dimensional position at the next moment is predicted based on the current trajectory: By combining the projection matrices of the left and right cameras, the predicted 3D points are reprojected onto the left and right image planes, respectively. This gives the predicted image point position of the measurement point in the left and right images in the next frame.

[0102] Example 2

[0103] See Figure 5 The present invention also provides a large-scale structure three-dimensional full-field deformation measurement image processing system, which is used in the large-scale structure three-dimensional full-field deformation measurement image processing method described in any one of the above claims, the system comprising:

[0104] The image sequence construction module 100 is used to construct a stereo image sequence; wherein, the first frame of the stereo image sequence is a reference stereo image of the structure under test in a reference state, the second frame of the stereo image and the subsequent multiple frames of the stereo image are multiple sets of dynamic stereo images of the structure under test in a load state, and the stereo image includes a left camera image and a right camera image.

[0105] The matching point search module 200 is used to select multiple measurement points in the left camera image of each frame of stereo image, and obtain the epipolar function relationship of each measurement point in the left camera image in the corresponding right camera image, so as to construct a one-dimensional search region in the right camera image; and use the box filtering integer pixel fast correlation search method to search for the matching point corresponding to each measurement point in the one-dimensional search region.

[0106] The three-dimensional vector calculation module 300 is used to measure the three-dimensional state vector of each measurement point at each frame time by using the bi-target positioning parameters and projection matrix, combined with the position parameters of the measurement points and the position parameters of the corresponding matching points.

[0107] The three-dimensional vector prediction module 400 is used to predict the state vector of each measurement point at the current frame time based on the three-dimensional state vector of each measurement point at the previous frame time using the Kalman filter method, so as to obtain the three-dimensional predicted state vector corresponding to the current frame time.

[0108] The 3D vector update module 500 is used to calculate the difference between the 3D predicted state vector and the 3D state vector at the current frame time using the correlation cost function, and update the 3D state vector at the current frame time based on the difference value.

[0109] The image fitting processing module 600 is used to update the three-dimensional state vectors of all measurement points at different frame times, and then use the three-dimensional state vectors at all frame times to generate the three-dimensional motion trajectory of the measurement points; and to perform spatial fitting on the three-dimensional motion trajectory of all measurement points to obtain the corrected image of the structure under test.

[0110] It should be noted that the modules in the system of Embodiment 2 correspond to the steps in the method of Embodiment 1. The steps in the method of Embodiment 1 have been described in detail in Embodiment 1, and the module content in the system will not be described in detail in this Embodiment 2.

[0111] Example 3

[0112] See Figure 6 This embodiment also provides a computer device, including a system memory 1005 and a processor 1001. The system memory 1005 stores a computer program, and the processor 1001 executes the computer program to implement the steps of any of the methods described above.

[0113] It should be noted that the processor 1001 is used to execute the steps in the above method embodiments according to the instructions in the program code. Alternatively, when the processor 1001 executes the computer program, it implements the functions of each module / unit in the above system / device embodiments.

[0114] Specifically, in this embodiment, the computer program can be divided into one or more modules / units, which are stored in the system memory 1005 and executed by the processor 1001 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device.

[0115] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor 1001 and a system memory 1005. Those skilled in the art will understand that this does not constitute a limitation on the terminal device; it may include more or fewer components than shown in the figures, or a combination of certain components, or different components. For example, the terminal device may also include an input / output device 1003, a network access device 1002, a bus 1006, etc.

[0116] The processor 1001 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0117] System memory 1005 can be an internal storage unit of the terminal device, such as a hard drive or RAM. System memory 1005 can also be a storage device 1004 of the terminal device, such as an external hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or FlashCard. Furthermore, system memory 1005 can include both internal storage units and storage device 1004. System memory 1005 is used to store computer programs and other programs and data required by the terminal device. System memory 1005 can also be used to temporarily store data that has been output or will be output.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0119] Example 4

[0120] This embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0121] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), registers, hard disks, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, or any other form of computer-readable storage medium in the art.

[0122] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). In embodiments of the invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device.

[0123] Example 5

[0124] This embodiment also provides a computer program product containing instructions that, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method described in Embodiment 1.

[0125] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for image processing of large-scale structural three-dimensional full-field deformation measurement, characterized in that, The methods include: Construct a stereo image sequence; wherein, the first frame of the stereo image sequence is a reference stereo image of the structure under test in a reference state, the second frame of the stereo image and the subsequent multiple frames of the stereo image are multiple sets of dynamic stereo images of the structure under test in a load state, and the stereo image includes a left camera image and a right camera image; Multiple measurement points are selected in the left camera image of each frame of stereo image, and the epipolar function relationship of each measurement point in the left camera image in the corresponding right camera image is obtained to construct a one-dimensional search region in the right camera image; and the integer pixel fast correlation search method of box filtering is used to search for the matching point corresponding to each measurement point in the one-dimensional search region. By using the bi-target positioning parameters and projection matrix, combined with the position parameters of the measurement points and the position parameters of the corresponding matching points, the three-dimensional state vector of each measurement point at each frame time is obtained. Based on the three-dimensional state vector of each measurement point in the previous frame, the Kalman filter method is used to predict the state vector of each measurement point in the current frame, so as to obtain the three-dimensional predicted state vector corresponding to the current frame. The difference between the 3D predicted state vector and the 3D state vector at the current frame time is calculated using the correlation cost function, and the 3D state vector at the current frame time is updated based on the difference value. After updating the three-dimensional state vectors of all measurement points at different frame times, the three-dimensional motion trajectory of the measurement points is generated using the three-dimensional state vectors at all frame times; spatial fitting is performed on the three-dimensional motion trajectory of all measurement points to obtain the corrected image of the structure under test.

2. The image processing method for large-scale structural three-dimensional full-field deformation measurement according to claim 1, characterized in that, Constructing a stereo image sequence is specifically as follows: A binocular vision measurement unit was constructed using two image acquisition devices, and the binocular vision measurement unit was calibrated. A reference binocular image is obtained by acquiring reference left and reference right camera images of the structure under test in a reference state using a binocular vision measurement unit. Using a binocular vision measurement unit, multiple dynamic left and right camera images of the structure under load are acquired at preset time intervals to obtain multiple sets of dynamic binocular images. The reference stereo image is used as the first frame of the stereo image in the sequence. Multiple sets of dynamic stereo images are arranged after the first frame of the stereo image according to the acquisition time sequence to construct the stereo image sequence.

3. The image processing method for large-scale structural three-dimensional full-field deformation measurement according to claim 1, characterized in that, In each frame of the stereo image, multiple measurement points are selected in the left camera image, specifically: In each frame of the binocular image, select the region of interest of the structure under test, divide the region of interest according to the preset region division rules, and take the center point of each relevant sub-region as the measurement point.

4. The image processing method for large-scale structural three-dimensional full-field deformation measurement according to claim 1, characterized in that, A fast integer-pixel correlation search method using box filtering is employed to find matching points corresponding to each measurement point within a one-dimensional search region. Specifically: Each measurement point is located in a sub-region as a reference sub-region. The gray value of each reference sub-region is obtained, and the gray sum and gray square sum of the reference sub-region are calculated. In the direction of the one-dimensional search area corresponding to each measurement point, the pre-constructed search window is moved according to the preset moving step size, and the area after each search window movement is taken as a candidate sub-region. The box filtering incremental update strategy is used to obtain the gray sum, gray square sum and gray product sum between the candidate sub-region and the corresponding reference sub-region after each movement. The sum of gray levels, the sum of squared gray levels, and the sum of gray-level products of each candidate sub-region are compared with the sum of gray levels and the sum of squared gray levels of the corresponding reference sub-region. These are then input into the cross-correlation coefficient calculation function to calculate the correlation coefficient between each candidate sub-region and the reference sub-region. The center point of the candidate sub-region with the highest correlation coefficient is selected as the matching point of the corresponding measurement point.

5. The image processing method for large-scale structural three-dimensional full-field deformation measurement according to claim 4, characterized in that, After obtaining the corresponding matching point at each measurement point, the method also includes: Based on the peak correlation coefficient and neighborhood correlation value of the matching point, the position of the matching point is corrected by a quadratic polynomial curve fitting method to obtain the corrected position of the matching point.

6. The image processing method for large-scale structural three-dimensional full-field deformation measurement according to claim 1, characterized in that, The 3D state vector at the current frame time is updated based on the difference value, specifically as follows: When the difference value is not less than the preset threshold, the three-dimensional predicted state vector is replaced with the three-dimensional state vector at the current frame time. When the difference value is less than the preset threshold, the three-dimensional state vector will be retained at the current frame time.

7. A large-scale structural three-dimensional full-field deformation measurement image processing system, characterized in that, This system is used in the image processing method for measuring the three-dimensional full-field deformation of a large-scale structure as described in any one of claims 1-6, wherein the system comprises: An image sequence construction module is used to construct a stereo image sequence; wherein, the first frame of the stereo image sequence is a reference stereo image of the structure under test in a reference state, and the second frame of the stereo image and the subsequent multiple frames of the stereo image are multiple sets of dynamic stereo images of the structure under test in a load state, and the stereo image includes a left camera image and a right camera image. The matching point search module is used to select multiple measurement points in the left camera image of each frame of stereo image and obtain the epipolar function relationship of each measurement point in the left camera image in the corresponding right camera image to construct a one-dimensional search region in the right camera image; and adopts the box filtering integer pixel fast correlation search method to search for the matching point corresponding to each measurement point in the one-dimensional search region. The three-dimensional vector calculation module is used to measure the three-dimensional state vector of each measurement point at each frame time by using the bi-target positioning parameters and projection matrix, combined with the position parameters of the measurement points and the position parameters of the corresponding matching points. The 3D vector prediction module is used to predict the state vector of each measurement point in the current frame based on the 3D state vector of each measurement point in the previous frame, using the Kalman filter method, so as to obtain the 3D predicted state vector corresponding to the current frame. The 3D vector update module is used to calculate the difference between the 3D predicted state vector and the 3D state vector at the current frame time using the correlation cost function, and update the 3D state vector at the current frame time based on the difference value. The image fitting processing module is used to update the three-dimensional state vectors of all measurement points at different frame times, and then use the three-dimensional state vectors at all frame times to generate the three-dimensional motion trajectory of the measurement points; and to perform spatial fitting on the three-dimensional motion trajectory of all measurement points to obtain the corrected image of the structure under test.

8. A computer device comprising a system memory and a processor, wherein the system memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 6.

10. A computer program product containing instructions, characterized in that, When the instruction is executed by a cluster of computer devices, the cluster of computer devices causes the cluster of computer devices to perform the method as described in any one of claims 1 to 6.

Images