Full-automatic pixel-by-pixel output three-dimensional digital image correlation method based on deep learning

Through the deep learning-based displacement estimation model, combined with GMA and RAFT-DIC sub-models, the problem of difficulty in manual parameter selection and large displacement measurement in traditional 3D-DIC is solved, and a fully automatic, pixel-by-pixel three-dimensional morphology and displacement field are realized Measurement improves measurement accuracy and robustness.

CN118918245APending Publication Date: 2024-11-08BEIHANG UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410928981.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional 3D-DIC based on image sub-region cannot realize fully automatic, pixel-by-pixel deformation measurement without manual adjustment parameters, and cannot effectively deal with large displacement caused by parallax, resulting in limited morphology of the measurement object.

Method used

Using a deep learning-based displacement estimation model, including the GMA sub-model and the RAFT-DIC sub-model, the parallax displacement field is calculated through stereo matching and timing matching, and combined with the camera internal and external parameters of the measurement system, the three-dimensional morphology of the object surface is reconstructed and Displacement field realizes high-precision measurement of large displacement.

Benefits of technology

It realizes fully automatic deformation field output without manual parameter selection, avoids information loss caused by discrete calculation points, and can output dense displacement fields by pixel, solves the problem of large displacement measurement failure, and improves measurement accuracy and robustness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918245A_ABST
    Figure CN118918245A_ABST
Patent Text Reader

Abstract

The invention provides a full-automatic and pixel-by-pixel output three-dimensional digital image correlation method based on deep learning. On the basis of traditional three-dimensional reconstruction and displacement field calculation based on binocular stereoscopic vision, the method has the advantages that the method is simple; a deep learning displacement calculation framework comprising two sub-models is provided to replace a digital image correlation method based on an image sub-region, and stereo matching and time sequence matching of images are realized. For the deep learning framework, the large displacement estimation sub-model firstly calculates a rough displacement field, distorts the displacement field to deform an image, and eliminates main displacement components. And then the high-precision small-displacement estimation sub-model further extracts relatively small residual deformation between the reference image and the distorted image. And finally, subtracting the large displacement from the small displacement to obtain a final refined high-precision large-range displacement field. According to the invention, a deep learning displacement calculation framework is utilized, complex manual parameter selection is not needed, the advantage of pixel-by-pixel deformation field output can be realized, and full-automatic and pixel-by-pixel three-dimensional digital image correlation measurement is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a three-dimensional digital image correlation method with full-automatic, pixel-by-pixel output based on deep learning. Background Art

[0002] Three-dimensional digital image correlation (3D-DIC) is an optomechanical technique for non-contact, full-field three-dimensional topography and deformation measurement. Due to its outstanding advantages such as simple optical setup, easy implementation, low requirements for experimental environment, adjustable spatiotemporal resolution, and wide applicability, it has irreplaceable and extensive applications in various scientific disciplines and engineering fields. 3D-DIC can be divided into two subtasks: (1) stereo matching for reconstructing the three-dimensional topography of the object surface; (2) temporal matching for tracking the deformation of the object surface. For a given 3D-DIC experimental data, the key issue is to determine the basic three-dimensional topography and displacement field of the test object surface. The standard sub-area-based DIC algorithm works by subdividing the input reference image into small sub-areas, followed by cross-correlation calculations between consecutive images to retrieve the displacement vectors at the centers of the image sub-areas.

[0003] Although image sub-area-based DIC has made significant progress in measurement accuracy, computational efficiency, and robustness, it still has some unresolved issues: 1) DIC analysis requires users to input key calculation parameters (sub-area size, step size, and shape function order, etc.) to perform relevant analysis. In practice, selecting appropriate parameters is tricky, especially for complex shapes or unknown deformation fields. In this case, inappropriate calculation parameters will lead to erroneous calculation results. 2) It is impossible to calculate the three-dimensional morphology and deformation field pixel by pixel. The existing image sub-area-based DIC calculations are performed at uniformly distributed calculation points specified on the reference image. The reconstructed morphology and displacement field show the characteristics of a low-pass filtering effect, resulting in a serious loss of reconstructed details.

[0004] To address these inherent limitations of image sub-area-based 3D-DIC, deep learning is a possible solution. Deep learning-based DIC can achieve fully automatic displacement field output without manual parameter selection. In addition, the deep learning displacement calculation framework can output dense pixel-by-pixel displacement fields, avoiding information loss caused by discrete calculation points. However, most existing deep learning displacement calculation frameworks can only be applied to two-dimensional digital image correlation measurements. For 3D-DIC, three-dimensional reconstruction requires stereo matching of images captured by the camera from two perspectives, but the displacement caused by parallax is often tens or hundreds of pixels, which is far beyond the displacement measurement range of existing deep learning displacement calculation frameworks (usually within 10 pixels). Therefore, the deep learning displacement calculation framework cannot be directly applied to 3D-DIC.

[0005] There are many related studies that extend deep learning to 3D-DIC. For example, StrainNet-3D uses affine transformation to calculate disparity and uses a lightweight model to calculate small displacement fields. However, the captured surface must satisfy an approximate affine transformation relationship between the left and right images. This imposes strict restrictions on the shape of the object being measured (i.e., the object morphology must be quasi-planar). Recently, NAS-Net established a virtual 3D-DIC measurement system using determined internal and external parameters. On this basis, a training dataset is generated to train the displacement estimation model and achieve end-to-end displacement field output. Experimental results demonstrate the effectiveness of NAS-Net. However, the morphology of the object being measured is also limited to quasi-planes. In addition, the model is trained on a dataset within a virtual system with specific internal and external parameters. When the measurement system changes, the dataset needs to be regenerated and the model needs to be retrained.

[0006] In summary, traditional 3D-DIC based on image sub-areas requires manual parameter selection, cannot achieve fully automatic measurement, and cannot achieve pixel-by-pixel deformation field calculation. The development of existing deep learning-based 3D-DIC technology is still in its early stages. The key problem is that the large displacement caused by parallax cannot be solved, and the stereo matching of 3D-DIC cannot be effectively achieved, which limits the morphology of the measured object. In order to achieve fully automatic, pixel-by-pixel output of 3D-DIC and avoid the problem of failure of large displacement measurement of existing deep learning 3D-DIC, a deep learning displacement calculation framework that can achieve large displacement measurement is urgently needed for stereo matching and time series matching of images. Summary of the invention

[0007] The purpose of the present invention is to provide a 3D-DIC deformation measurement method based on deep learning to overcome the problem that traditional image sub-area based 3D-DIC cannot achieve fully automatic, pixel-by-pixel deformation measurement without manual adjustment of parameters.

[0008] To achieve the above object, the present invention provides a fully automatic, pixel-by-pixel output 3D-DIC deformation measurement method based on deep learning, comprising:

[0009] Get the left and right images of the object surface at the same time;

[0010] Performing stereo matching on the left and right images to calculate the parallax displacement field between the left and right views;

[0011] Perform time sequence matching on the left and right images at different times to obtain the displacement field of time sequence matching;

[0012] Reconstructing the three-dimensional morphology of the object surface at different stages based on the parallax displacement field between the left and right views and combining the calibration results of the internal and external parameters of the camera in the measurement system;

[0013] For the point cloud coordinates of the three-dimensional shape in the initial state, the corresponding point cloud coordinates of the three-dimensional shape in the deformed state are subtracted according to the result of the time series matching to calculate the three-dimensional full-field displacement field in the deformed state.

[0014] In particular, a displacement estimation model based on deep learning is used to perform stereo matching and temporal matching on the left and right images; wherein the displacement estimation deep learning model includes: a GMA sub-model and a RAFT-DIC sub-model.

[0015] In particular, the deep learning displacement estimation model used includes the following steps:

[0016] A pair of reference images taken at the same time and a pair of deformed images taken during the deformation phase are input into the GMA sub-model to output a rough displacement field with a large displacement range;

[0017] Distorting the deformed image based on the rough displacement field to obtain a distorted deformed image with large displacement components eliminated;

[0018] Inputting the reference image and the distorted image into the RAFT-DIC sub-model, and outputting a compensated small displacement field;

[0019] The coarse displacement field is subtracted from the compensated small displacement field to obtain a final refined displacement field.

[0020] Optionally, the reference image and the distorted image are input into the RAFT-DIC sub-model, and a compensated small displacement field is output;

[0021] Input the reference image and the distorted image into a feature encoder network to extract feature information, and output corresponding reference image features w1 and distorted image features w2;

[0022] Calculate the inner product between the reference image feature w1 and the deformed image feature w2 to obtain a 4D correlation layer;

[0023] The 4D correlation layer, the deformed image feature w2 and the initial displacement field are input into the iterative update operator based on Conv-GRU, and the estimated displacement field is iteratively updated to obtain the final compensated small displacement field.

[0024] Optionally, the feature encoder network includes a plurality of residual blocks, with channels of 64, 128, 192 and 256 respectively;

[0025] The feature encoder network also includes a context encoding network, and the additional features of the reference image are extracted based on the context encoding network.

[0026] Optionally, the 4D correlation quantity vector pair is:

[0027]

[0028] Among them, i and j represent the coordinates in the reference image, k and l represent the coordinates in the deformed image, and h represents the h of the extracted reference image feature w1 and the distorted deformed image feature w2. th Channel, C ijkl Represents a 4D correlation layer vector pair.

[0029] Optionally, the iterative update operator based on Conv-GRU includes four input parts and outputs a new hidden state, and the output hidden state is input into two convolutional layers to obtain a residual displacement field; wherein the input part includes: a displacement field, a related layer, a hidden state and context features.

[0030] Optionally, the method to update the displacement field estimated by RAFT-DIC is:

[0031] f k+1 =Δf+f k

[0032] Among them, f k+1 represents the displacement field output by RAFT-DIC after the kth iteration, Δf represents the residual displacement field, and f k represents the displacement field output by RAFT-DIC after the k+1th iteration.

[0033] The present invention has the following beneficial effects:

[0034] The present invention proposes a 3D-DIC deformation measurement method based on deep learning. Compared with the traditional 3D-DIC method based on image sub-areas, the present invention can realize fully automatic deformation field output without complex manual parameter selection. In addition, the present invention can output pixel-by-pixel dense displacement fields, avoiding the loss of reconstruction information caused by discrete calculation points. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 A schematic diagram of a three-dimensional digital image correlation method based on deep learning according to an embodiment of the present invention;

[0037] Figure 2 This is a flow chart of a three-dimensional digital image correlation method based on deep learning according to an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the deep learning-based displacement estimation model G-RAFT according to an embodiment of the present invention;

[0039] Figure 4 Schematic diagram of the sub-model RAFT-DIC of the displacement estimation network G-RAFT according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] This embodiment proposes a fully automatic, pixel-by-pixel output three-dimensional digital image correlation method based on deep learning, including:

[0043] Stereo matching is performed on the left and right images of the object surface at the same time to predict the disparity between the left and right views;

[0044] Perform time-series matching on the left and right images at different times to obtain the in-plane deformation information of the object surface;

[0045] Reconstruct the 3D shape of the object surface at different stages based on the parallax between the left and right views;

[0046] The 3D full-field displacement field of the deformed state is calculated by subtracting the 3D coordinates of the deformed state from the initial state.

[0047] In particular, for the above-mentioned stereo matching and temporal matching, a displacement estimation model based on deep learning is used; wherein the displacement estimation deep learning model includes: a GMA sub-model and a RAFT-DIC sub-model.

[0048] Furthermore, based on the displacement estimation deep learning model, stereo matching is performed on the left and right images of the object surface at the same time, including:

[0049] A pair of reference images taken at the same time and a pair of deformed images taken during the deformation phase are input into the GMA sub-model, and a rough displacement field over a large range is output;

[0050] Distorting the deformed image based on the rough displacement field to obtain a distorted deformed image with large displacement components eliminated;

[0051] The reference image and the distorted image are input into the RAFT-DIC sub-model, and the compensated small displacement field is output;

[0052] The rough displacement field is added to the compensated small displacement field to obtain the final refined displacement field.

[0053] Furthermore, the reference image and the distorted image are input into the RAFT-DIC sub-model, and the output compensated small displacement field includes:

[0054] Input the reference image and the distorted image into the feature encoder network to extract feature information, and output the corresponding reference image feature w1 and distorted image feature w2;

[0055] Calculate the inner product between the reference image feature w1 and the deformed image feature w2, obtain a 4D correlation layer, and calculate the similarity between the reference image feature w1 and the deformed image feature w2. The 4D correlation layer will be further input into the subsequent iterative update operator.

[0056] The 4D correlation layer, the distorted image feature w2 and the initial displacement field (the initial displacement field value is 0) are input into the iterative update operator based on Conv-GRU, and the estimated displacement field is updated by iterative RAFT-DIC to obtain the final compensated small displacement field.

[0057] Furthermore, the feature encoder network includes several residual blocks with 64, 128, 192, and 256 channels;

[0058] The feature encoder network also includes a context encoding network, which extracts additional features of the reference image based on the context encoding network to provide more image information to the Conv-GRU based iterative update operator.

[0059] Furthermore, the iterative update operator based on Conv-GRU includes four input parts and outputs a new hidden state. The output hidden state is input into two convolutional layers to obtain the residual displacement field; wherein the input part includes: displacement field, related layer, hidden state and context feature.

[0060] The present embodiment proposes a three-dimensional digital image correlation method based on deep learning and outputted pixel by pixel, including: stereo calibration of the measurement system; stereo matching of the left reference image and the right reference image, and temporal matching of all left and right deformed images using a displacement calculation framework based on deep learning; reconstructing the three-dimensional morphology corresponding to each stage by combining the calibration results and disparity data; and calculating the three-dimensional full-field displacement field of the three-dimensional coordinates of each stage. In particular, the present embodiment proposes a displacement calculation framework based on deep learning called G-RAFT, which is used for displacement field calculation of stereo matching and temporal matching, and contains two sub-models for large displacement estimation and small displacement estimation, respectively. Specifically, a large displacement estimation model GMA is used to calculate a large-scale rough displacement field, and the rough displacement field is distorted on the deformed image to eliminate the main displacement component. The small residual deformation between the reference image and the distorted image is further extracted using the high-precision small displacement estimation model RAFT-DIC. The calculated large displacement and small displacement are subtracted to obtain the final refined high-precision large-scale displacement field. This embodiment realizes fully automatic, pixel-by-pixel three-dimensional digital image correlation measurement through a displacement calculation framework based on deep learning.

[0061] Specifically, Figure 1-Figure 2 As shown, the detailed steps of this embodiment are:

[0062] Step 1: Input the left and right images at the same time into the displacement estimation deep learning model for stereo matching, calculate the disparity between the left and right views, and input the left and right images at different times into the displacement estimation deep learning model for time series matching to obtain the deformation information of the object surface; the left and right images are captured by a binocular camera;

[0063] Before using a binocular camera to capture the speckle images of the specimen before and after deformation, the binocular camera needs to be calibrated. The calibration results are used for stereo correction of the left and right images and three-dimensional reconstruction of the object surface.

[0064] Step 2: Based on binocular stereo vision theory, the parallax between the left and right views is corrected to reconstruct the three-dimensional appearance of the object surface at different stages;

[0065] Step 3: Further calculate the three-dimensional full-field displacement field at different deformation stages by subtracting the three-dimensional coordinates at different deformation stages from the initial state.

[0066] The displacement estimation deep learning model in step 1 is as follows: Figure 3As shown, specifically, a displacement estimation model called G-RAFT is constructed, which includes two sub-models, GMA and RAFT-DIC, and combines the coarse-fine strategy to achieve high-precision matching of large displacements. GMA is an advanced model in the field of optical flow estimation, with excellent global matching capabilities, but its accuracy cannot meet the requirements of DIC measurement. RAFT-DIC is based on the improvement of the RAFT framework in the field of optical flow, which significantly improves the measurement accuracy and robustness of the original framework, but the displacement range is insufficient and cannot directly handle large displacements caused by parallax. The details are as follows:

[0067] A pair of reference and deformed images is input into the GMA model, which outputs a rough displacement field with a large displacement range.

[0068] The coarse displacement field is warped on the deformed image to obtain a distorted deformed image with large displacement components eliminated. The residual deformation between the reference image and the distorted deformed image is greatly reduced.

[0069] The reference image and the distorted image are input into RAFT-DIC, and the compensated small displacement field is output.

[0070] The rough displacement field obtained by GMA was subtracted from the compensated displacement field obtained by RAFT-DIC to obtain the final refined displacement field.

[0071] The RAFT-DIC framework has the following improvements compared to the original RAFT framework:

[0072] In the RAFT-DIC feature extraction module, in order to perceive more contextual information of the speckle image, all step sizes are set to 1, so no downsampling occurs and the output feature resolution is the same as the image resolution.

[0073] The resolution of the last two dimensions of the correlation volume is reduced by using pooling kernels of size 1 and 2, resulting in full-resolution and 1 / 2 resolution quads and a two-level correlation pyramid.

[0074] Specifically, Figure 4 As shown, the detailed description of the RAFT-DIC model is as follows;

[0075] The reference image I1 and the deformed image I2 are input into the feature encoder network to extract feature information and output two corresponding features w1 and w2. The feature encoder consists of six residual blocks with channels of 64, 128, 192 and 256 respectively. In order to perceive more contextual information of the speckle image, an additional network called context encoding is used to extract additional features of the reference image I1. The context encoder has the same architecture as the feature encoder, but different parameters. In the overall feature extraction module, all convolution steps are 1, so no downsampling occurs, and the output feature resolution is the same as the image resolution.

[0076] The correlation layer is used to calculate the similarity between the reference image feature w1 and the deformed image feature w2, and the 4D correlation quantity C(w1,w2) vector pair is calculated by calculating the inner product between the features, as shown below:

[0077]

[0078] Where i and j represent the coordinates in the reference image, k and l represent the coordinates in the deformed image, and h represents the h of the extracted image feature. th Channels. For an input image of size W×H, the correlation layer generates a 4D full correlation volume of size W×H×W×H. The correlation layer is essentially a feature matching process between features w1 and w2. To further enable RAFT-DIC to perceive larger displacements, the resolution of the last two dimensions of the correlation volume is reduced by pooling kernels of size 1 and 2, resulting in full-resolution and 1 / 2 resolution 4D volumes and a two-layer correlation pyramid is constructed.

[0079] The estimated displacement field is updated using an iterative update operator based on Conv-GRU. The iterative update operator mainly consists of four input parts (displacement field, correlation layer, hidden state and context features), outputs a new hidden state, which is then input into two convolutional layers to obtain the residual displacement field Δf. When the update operator starts from the initial displacement field f0=0, it estimates a series of estimated displacement fields {f0,…,f N}, the updating process of the displacement field can be described as:

[0080] f k+1 =Δf+f k

[0081] Among them, f k+1 represents the displacement field output by RAFT-DIC after the kth iteration, Δf represents the residual displacement field, and f k represents the displacement field output by RAFT-DIC after the k+1th iteration.

[0082] The final output displacement field of RAFT-DIC is f k+1 .

[0083] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A fully automatic, pixel-by-pixel output 3D digital image correlation method based on deep learning, It is characterized in that include: Get the left and right images of the object surface at the same time; Performing stereo matching on the left and right images to predict the disparity between the left and right views; Perform time sequence matching on the left and right images at different times to obtain the result of time sequence matching; Reconstructing the three-dimensional appearance of the object surface at different stages based on the parallax between the left and right views; For the point cloud coordinates of the three-dimensional shape in the initial state, the corresponding point cloud coordinates of the three-dimensional shape in the deformed state at different stages are subtracted according to the result of the time series matching to calculate the three-dimensional full-field displacement field at different deformation stages.

2. The fully automatic, pixel-by-pixel output three-dimensional digital image correlation method based on deep learning according to claim 1, It is characterized in that The left and right images are stereo matched using a depth-based learning displacement estimation model; wherein the displacement estimation deep learning model includes: a GMA sub-model and a RAFT-DIC sub-model.

3. The fully automatic, pixel-by-pixel output three-dimensional digital image correlation method based on deep learning according to claim 2, It is characterized in that Using a depth-based learning displacement estimation model, stereo matching is performed on the left and right images, including: A pair of reference images taken at the same time and a pair of deformed images taken during the deformation phase are input into the GMA sub-model, and a rough displacement field over a large range is output; Distorting the deformed image based on the rough displacement field to obtain a distorted deformed image with large displacement components eliminated; Inputting the reference image and the distorted image into the RAFT-DIC sub-model, and outputting a compensated small displacement field; The coarse displacement field is subtracted from the compensated small displacement field to obtain a final refined displacement field.

4. The fully automatic, pixel-by-pixel output three-dimensional digital image correlation method based on deep learning according to claim 3, It is characterized in that Inputting the reference image and the distorted image into the RAFT-DIC sub-model, and outputting a compensated small displacement field; The reference image and the distorted image are input into the feature encoder network to extract feature information and output the corresponding reference image feature w 1 and the distorted image feature w 2 ; Calculate the reference image feature w 1 and the deformed image feature w 2 The inner product between them obtains the 4D correlation layer; The 4D correlation layer, the deformed image feature w 2 and the initial displacement field, and input them into the iterative update operator based on Conv-GRU, iteratively update the estimated displacement field, and obtain the final compensated small-scale displacement field.

5. The fully automatic, pixel-by-pixel output three-dimensional digital image correlation method based on deep learning according to claim 4, It is characterized in that The feature encoder network includes several residual blocks with channels of 64, 128, 192 and 256 respectively; The feature encoder network also includes a context encoding network, and the additional features of the reference image are extracted based on the context encoding network.

6. The fully automatic, pixel-by-pixel output three-dimensional digital image correlation method based on deep learning according to claim 4, It is characterized in that The 4D correlation vector pair is: Among them, i and j represent the coordinates in the reference image, k and l represent the coordinates in the deformed image, and h represents the extracted reference image feature w 1 and the distorted image feature w 2 h th Channel, C ijkl Represents a 4D correlation layer vector pair.

7. The fully automatic, pixel-by-pixel output three-dimensional digital image correlation method based on deep learning according to claim 4, It is characterized in that The iterative update operator based on Conv-GRU includes four input parts and outputs a new hidden state. The output hidden state is input into two convolutional layers to obtain a residual displacement field; wherein the input part includes: a displacement field, a related layer, a hidden state and context features.

8. The fully automatic, pixel-by-pixel output three-dimensional digital image correlation method based on deep learning according to claim 7, It is characterized in that The method to update the estimated displacement field is: f k+1 =Δf+f k Among them, f k+1 represents the displacement field output by RAFT-DIC after the kth iteration, Δf represents the residual displacement field, and f k represents the displacement field output by RAFT-DIC after the k+1th iteration.

Citation Information

Cited By

  • Three-dimensional DIC method based on sub-pixel cost body and uncertainty

    CN120147525A