Digital image correlation method based on deep learning image matching algorithm

Abstract

A digital image correlation method based on a deep learning image matching algorithm belongs to the field of deformation measurement.The method of the present application is as follows: speckle images before and after deformation of a sample are collected, the speckle image before deformation is a reference image, the speckle image after deformation is a target image, a calculation region and a calculation point are set in the reference image; a deep convolutional neural network-based image feature extraction network is used to extract key features of the reference image and the target image; a Transform-based image feature matching network is used to match the features to form feature point pairs; the incorrectly matched feature point pairs are removed, and an affine transformation is used to calculate a deformation initial value; based on the deformation initial value and an IC-GN algorithm, a final high-precision deformation field is calculated by nonlinear iteration, and the digital image correlation is realized based on the deep learning image matching algorithm.The present application matches speckle features by using the deep learning image matching algorithm, and can realize deformation measurement of low-quality speckle images.

Description

Technical Field

[0001] The present invention relates to a digital image correlation method based on a deep learning image matching algorithm, and belongs to the field of deformation measurement. Background Art

[0002] Accurately measuring material deformation information is crucial for evaluating material properties, optimizing design, monitoring structural safety, predicting service life, and formulating relevant standards. It is an indispensable part of experimental mechanics, materials science and engineering.

[0003] Traditional deformation measurement methods primarily rely on contact-based measurements, such as strain gauges and extensometers. These methods require the strain gauge or extensometer to be fixed to the test area, allowing it to deform along with the specimen. The deformation is then calculated based on the change in resistance of the conductive material within. However, these methods can only measure point deformation on the material surface or the average deformation of a localized area of ​​the specimen, failing to capture strain field information.

[0004] Digital Image Correlation (DIC) is a widely used non-contact deformation measurement method. Its basic principle is to generate speckles on the material surface, or to generate speckles using digital radiography (DR) based on the material's internal texture. By comparing the digital speckle images before and after deformation of the specimen, the deformation field information of the speckles is measured, thereby reflecting the deformation field information of the specimen. The DIC method consists of three main steps: initial deformation estimation, nonlinear iterative optimization, and strain calculation. The accuracy of the initial deformation estimation determines whether the subsequent iterative optimization can converge to an accurate result.

[0005] The DIC method measures deformation based on speckle images, so the quality of the speckle image is significantly correlated with the accuracy of the measurement results. However, in high-temperature experiments or when using DR images for deformation field measurement, the speckle image degrades, with reduced contrast and increased noise. Traditional methods for estimating initial deformation values ​​are prone to mismatching in these situations, failing to find the corresponding subregions in the speckle images before and after deformation, leading to calculation failure. Summary of the Invention

[0006] In order to solve the problem that the deformation cannot be measured under the condition of poor speckle image quality, the purpose of the present application is to provide a digital image correlation method based on a deep learning image matching algorithm; the method uses deep learning technology to automatically extract image features, and uses an efficient feature matching network for feature matching, calculates the deformation initial value of the point according to the matched feature points around the point, and then uses the inverse composition Gauss-Newton algorithm (IC-GN) to calculate the high-precision deformation field, and realizes digital image correlation based on the deep learning image matching algorithm.

[0007] The purpose of the present application is realized by the following technical solutions:

[0008] The present application discloses a digital image correlation method based on a deep learning image matching algorithm, comprising the following steps:

[0009] S1: collect speckle images before and after deformation of the sample, the speckle image before deformation is a reference image, the speckle image after deformation is a target image, and a calculation region and a calculation point are set in the reference image;

[0010] S2: use an image feature extraction network based on a deep convolutional neural network to extract key features of the reference image and the target image;

[0011] S3: use an image feature matching network based on a Transformer to match the key features of the reference image and the target image in step S2, and form a feature point pair;

[0012] S4: extract the feature point pairs within a preset range around the calculation point in step S1 from the feature point pairs obtained in S3, eliminate the incorrectly matched feature point pairs, and use affine transformation to calculate the deformation initial value of the calculation point;

[0013] S5: for the calculation points in S4 that cannot calculate the deformation initial value due to insufficient number of feature point pairs, use the deformation initial value of the nearest other calculation point as the initial value of the calculation point according to the continuity of displacement;

[0014] S6: based on the deformation initial values obtained in steps S4 and S5, use the inverse composition Gauss-Newton algorithm (IC-GN) to calculate the final high-precision deformation field through nonlinear iteration, that is, realize digital image correlation based on the deep learning image matching algorithm.

[0015] Further, in step S1, the collected image refers to the image of the speckle that deforms simultaneously with the sample, and the method of collecting the image of the deformed speckle includes: using an optical camera to collect the speckle image on the surface of the sample or using an X-ray imaging device to collect the projection image of the sample.

[0016] Further, in step S1, the speckle image refers to the speckle with uneven gray scale formed by artificially making speckles on the surface of the sample and relying on the rough surface of the material, and the speckle formed on the projection image by the multi-phase material relying on the internal natural texture.

[0017] Further, in step S1, the calculation area and the calculation point are set according to actual needs, and the calculation area is preferably a rectangular area determined by setting the coordinates of the upper left corner and the lower right corner, and the calculation points are uniformly distributed in the calculation area.

[0018] Further, in step S2, the structure of the deep convolutional neural network includes a shared encoder and two decoders; the shared encoder includes eight convolutional layers and three maximum pooling layers, wherein the size of the convolution kernel is 3x3, the step of the pooling layer is 2, and when the length and width of the input image are W and H respectively, the tensor dimension output by the shared encoder is (W / 8, H / 8, 128); the feature point decoder includes two convolutional layers, a Softmax function and a dimension transformation, wherein the size of the convolution kernel is 3x3 and 1x1, and the dimension of the tensor output by the shared encoder after passing through the feature point decoder is (W, H, 1), the value of each position represents the probability that the pixel is a feature point; the descriptor decoder includes two convolutional layers, an interpolation function and an L2 normalization function, wherein the size of the convolution kernel is 3x3 and 1x1, and the dimension of the tensor output by the shared encoder after passing through the descriptor decoder is (W, H, 256); the loss function of the feature extraction network includes feature point loss and descriptor loss.

[0019] Further, in step S2, the method for extracting key features is to screen the probability matrix output by the network feature point decoder, and the pixels with a probability greater than n are feature points, and the descriptors corresponding to the feature points are obtained through the descriptor decoder.

[0020] Further, in step S3, the structure of the image feature matching network described is composed of multiple layers, each layer is composed of a self-attention unit and a cross-attention unit, and a confidence classifier is used to determine whether to stop reasoning, and if the reasoning continues, the unmatched feature points are discarded to reduce the image feature matching calculation amount and improve the efficiency of image feature matching.

[0021] Further, in step S3, the matching process described is to input the feature points and descriptors of the reference image and the target image output in S2 into the image feature matching network, and output the matched feature point pairs after network reasoning.

[0022] Further, in step S3, the image feature matching network based on Transformer is used to match the features, and the matched feature point pairs are output.

[0023] Further, in step S4, a rectangular region is determined by setting the length and width of the predetermined range around the center point;

[0024] Further, in step S4, the implementation method of eliminating the error matching feature point pairs is as follows:

[0025] Step S4.1, three groups are randomly selected from the feature point pairs around the center point, the affine transformation parameters are calculated, and the initial affine transformation model is formed;

[0026] Step S4.2, all feature point pairs around the center point are fitted using the initial affine transformation model, if the calculation error of a certain pair of feature points is less than a certain threshold, it is considered to be an inner point pair, otherwise it is an outer point pair;

[0027] Step S4.3, if the number of inner point pairs exceeds a certain threshold, it is determined that the affine transformation model is reasonable, and the model parameters are re-estimated using all inner point pairs to improve the model; if the number of inner point pairs is less than the threshold, the elimination of error matching feature point pairs is restarted from step S4.1;

[0028] Further, in step S4, the affine transformation method refers to calculating the affine transformation parameters using the coordinates of the reference image feature points and the matching feature point coordinates in the target image, and then calculating the deformation initial value; a first-order affine transformation equation is selected:

[0029]

[0030] In the formula, (x, y) is the local coordinates of the key feature in the reference image relative to the center point, (x', y') is the local coordinates of the matching key feature in the target image relative to the center point, a 11 , a 12 , a 13 , a 21 , a 22 , a 23 are affine transformation parameters, more than three pairs of key features are required around the center point to solve the six parameters; the relationship between the affine transformation parameters and the deformation initial value is as follows:

[0031] a 11 = u x +1,

[0032] a 12 = u y ,

[0033] a 13 = u,

[0034] a 21 = v x ,

[0035] a 22 = vy +1,

[0036] a 23 =v.

[0037] wherein u, v represent the initial value of displacement of the calculation point along the x direction and the y direction, u x , u y represent the initial value of displacement of the calculation point along the x direction and the y direction, v x , v y represent the initial value of displacement of the calculation point along the x direction and the y direction, v

[0038] Advantages:

[0039] 1. The digital image correlation method based on the deep learning image matching algorithm disclosed in the present application automatically extracts image features by using deep learning technology, and performs feature matching by using an efficient feature matching network, calculates the deformation initial value of the point according to the matching feature points around the calculation point, and then calculates a high-precision deformation field by using an inverse combination Gauss-Newton algorithm IC-GN, realizes digital image correlation based on the deep learning image matching algorithm, automatically extracts features, and reduces the dependence on manual design of image matching.

[0040] 2. The digital image correlation method based on the deep learning image matching algorithm disclosed in the present application uses an image feature extraction network based on a deep convolutional neural network to extract key features of a reference image and a target image; the loss function of the feature extraction network comprises feature point loss and descriptor loss; the network is trained by using a deteriorated speckle image, so that the number and extraction efficiency of speckle image features are improved.

[0041] 3. The digital image correlation method based on the deep learning image matching algorithm disclosed in the present application uses an image feature matching network based on a Transformer to match the key features of the reference image and the target image to form a feature point pair; each layer is composed of a self-attention unit and a cross-attention unit, a confidence classifier is used to determine whether to stop reasoning, and if reasoning continues, the unmatched feature points are discarded, so as to reduce the image feature matching calculation amount and improve the efficiency and accuracy of image feature matching.

[0042] 4. The digital image correlation method based on the deep learning image matching algorithm disclosed in the present application considers the influence of noise and abnormal values on the network structure and training method of the image feature extraction network and the feature matching network, enhances the robustness of the deformation field of the digital image correlation measurement by improving the ability to match low-quality speckle images. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1A digital image correlation method flowchart based on a deep learning image matching algorithm;

[0044] Figure 2 A DR image of C / SiC composite material in an unloaded state;

[0045] Figure 3 A strain-time curve result graph calculated by the DIC method based on deep learning;

[0046] Figure 4 A strain-time curve result graph calculated by the open source program DIC program Ncorr;

[0047] Figure 5 A strain-time curve result graph calculated by the commercial software Vic 2D. DETAILED DESCRIPTION

[0048] In order to better illustrate the purposes and advantages of the present application, the content of the application will be further described below in combination with the drawings and examples.

[0049] Example 1:

[0050] As shown in Figure 1 , the present embodiment discloses a digital image correlation method based on a deep learning image matching algorithm, and the specific implementation steps are as follows:

[0051] S1: Use an X-ray imaging device to collect speckle images of a C / SiC composite material sample during deformation, and the speckle image before deformation is a reference image and the speckle image after deformation is a target image, relying on the natural texture inside the C / SiC composite material to form speckles in the image, as shown in Figure 2 . Set the calculation area and calculation points in the reference image. In the present embodiment, the calculation area size is 205*525px 2 , and the calculation point step is 5px.

[0052] S2: Use an image feature extraction network based on a deep convolutional neural network to extract key features of the reference image and the target image. The method for extracting key features is to filter the probability matrix output by the network feature point decoder, and the pixels with a probability greater than n are feature points, and the corresponding descriptors of the feature points are obtained through the descriptor decoder. In the present embodiment, n is set to 0.001.

[0053] The network structure of the image feature extraction network comprises a shared encoder and two decoders. The shared encoder comprises eight convolutional layers and three max-pooling layers, wherein the size of the convolution kernel is 3x3, and the step length of the pooling layer is 2. When the length and width of the input image are W and H respectively, the dimension of the tensor output by the shared encoder is (W / 8, H / 8, 128). The feature point decoder comprises two convolutional layers, a Softmax function and a dimension transformation, wherein the size of the convolution kernel is 3x3 and 1x1. The dimension of the tensor output by the shared encoder after passing through the feature point decoder is (W, H, 1), and the value of each position represents the probability that the pixel is a feature point. The descriptor decoder comprises two convolutional layers, an interpolation function and an L2 normalization function, wherein the size of the convolution kernel is 3x3 and 1x1. The dimension of the tensor output by the shared encoder after passing through the descriptor decoder is (W, H, 256). The loss function of the feature extraction network comprises a feature point loss and a descriptor loss.

[0054] S3: using a Transformer-based image feature matching network to match the features of the reference image and the target image in step S2 to form a feature point pair. The matching process is to input the feature points and descriptors of the reference image and the target image output in S2 into the image feature matching network, and output the matched feature points after network inference.

[0055] The network structure of the image feature matching network comprises multiple layers, each layer being composed of a self-attention unit and a cross-attention unit. A confidence classifier is used to determine whether to stop inference. If the inference continues, the unmatched feature points are discarded to reduce the amount of calculation and improve the calculation efficiency.

[0056] S4: extracting the feature point pairs within a range of 30pxx30px around the calculation point in step S1 from the feature point pairs obtained in S3, eliminating the incorrectly matched feature point pairs, and using affine transformation to calculate the deformation initial value of the calculation point.

[0057] In step S4, the implementation method of eliminating the incorrectly matched feature point pairs is as follows:

[0058] Step S4.1, randomly selecting three groups from the feature point pairs around the calculation point, calculating the affine transformation parameters, and forming an initial model.

[0059] Step S4.2, fitting all feature point pairs around the calculation point using the initial model. If the calculation error of a pair of feature points is less than a certain threshold, it is considered to be an internal point pair, otherwise it is an external point pair. In this embodiment, the threshold used is 1.

[0060] Step S4.3, if the number of inlier pairs exceeds a certain threshold, the model is considered to be reasonable, and all inlier pairs are used to re-estimate the model parameters to improve the model; if the number of inlier pairs is below the threshold, the process of eliminating the wrong matching feature point pairs is restarted from step S4.1.

[0061] The affine transformation method refers to calculating affine transformation parameters by using the coordinates of the reference image feature points and the coordinates of the matching feature points in the target image, and then calculating the deformation initial value. A first-order affine transformation equation is selected:

[0062]

[0063] In the formula, (x, y) is the local coordinates of the feature points in the reference image relative to the calculation point, (x', y') is the local coordinates of the matching feature points in the target image relative to the calculation point, a 11 , a 12 , a 13 , a 21 , a 22 , a 23 are affine transformation parameters, and more than three pairs of feature points around the calculation point are required to solve the six parameters. The relationship between the affine transformation parameters and the deformation initial value is as follows:

[0064] a 11 = u x + 1,

[0065] a 12 = u y ,

[0066] a 13 = u,

[0067] a 21 = v x ,

[0068] a 22 = v y + 1,

[0069] a 23 = v.

[0070] Wherein u, v represent the initial displacement of the calculation point along the x direction and the y direction, u x , u y represent the gradient initial value of the displacement of the calculation point along the x direction in the x and y directions, v x , v y represent the gradient initial value of the displacement of the calculation point along the y direction in the x and y directions.

[0071] S5: Based on the deformation initial value in step S4 and the IC-GN algorithm, the final high-precision deformation field is calculated. The average strain of the stretching direction calculation area of each target image in the deformation process is calculated to obtain the strain-time curve of the uniaxial stretching process of the C / SiC composite material, as shown in Figure 3 The strain-time curves obtained by using Ncorr and Vic 2D DIC software for calculation are shown in Figure 4 and Figure 5 The results are analyzed. Due to the characteristics of low contrast, periodicity and high noise of the DR image speckle of the C / SiC composite material, the deformation initial value calculation of Ncorr is prone to error, part of the results of the curve is missing, the results of Vic 2D fluctuate greatly, and the DIC method based on deep learning image matching algorithm has good robustness.

[0072] The above specific description further details the purpose, technical scheme and beneficial effects of the application. It should be understood that the above description is only a specific embodiment of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A digital image correlation method based on a deep learning image matching algorithm, characterized by: The following steps are included: S1: Collect speckle images of the sample before and after deformation. The speckle image before deformation is the reference image, and the speckle image after deformation is the target image. Set the calculation area and calculation points in the reference image. S2: Use an image feature extraction network based on a deep convolutional neural network to extract key features of the reference image and the target image; The structure of the deep convolutional neural network includes a shared encoder and two decoders; the shared encoder contains eight convolutional layers and three maximum pooling layers, wherein the size of the convolution kernel is 3×3, the stride of the pooling layer is 2, and when the length and width of the input image are W and H respectively, the dimension of the tensor output by the shared encoder is (W / 8, H / 8, 128); the feature point decoder contains two convolutional layers, a Softmax function and a dimensional transformation, wherein the size of the convolution kernel is 3×3 and 1×1, and the dimension of the tensor output by the shared encoder after passing through the feature point decoder is (W, H, 1), and the value of each position represents the probability that the position is a feature point; the descriptor decoder contains two convolutional layers, an interpolation function and an L2 normalization function, wherein the size of the convolution kernel is 3×3 and 1×1, and the dimension of the tensor output by the shared encoder after passing through the descriptor decoder is (W, H, 256); the loss function of the feature extraction network includes feature point loss and descriptor loss; In step S2, the method for extracting key features is to screen the probability matrix output by the network feature point decoder, and the pixels with a probability greater than n are feature points, and the descriptors corresponding to the feature points are obtained through the descriptor decoder; S3: Use the Transformer-based image feature matching network to match the key features of the reference image and the target image in step S2 to form feature point pairs; The structure of the image feature matching network described in step S3 includes a multi-layer network, each layer consists of self-attention units and cross-attention units, and uses a confidence classifier to determine whether to stop reasoning. If reasoning continues, unmatched feature points are discarded; The matching process described in step S3 is to input the feature points and descriptors of the reference image and the target image output in S2 into the image feature matching network, and output the matching feature point pairs after network inference; S4: Extract feature point pairs within a preset range around the calculation point in step S1 from the feature point pairs obtained in S3, remove incorrectly matched feature point pairs, and calculate the initial deformation value of the calculation point using affine transformation; S5: For the calculation point in S4 where the initial deformation value cannot be calculated due to insufficient number of feature point pairs, the initial deformation value of the nearest calculation point is used as the initial value of the calculation point according to the continuity of displacement; S6: Based on the initial deformation values ​​obtained in steps S4 and S5, the inverse combined Gauss-Newton algorithm IC-GN algorithm is used to nonlinearly iteratively calculate the final high-precision deformation field.

2. The digital image correlation method based on a deep learning image matching algorithm according to claim 1, characterized in that: In step S1, capturing an image refers to using an imaging device to capture an image of the speckle that deforms simultaneously with the sample. Methods for capturing the image of the deformed speckle include: using an optical camera to capture the speckle image on the sample surface or using an X-ray imaging device to capture a projection image of the sample.

3. The digital image correlation method based on a deep learning image matching algorithm according to claim 1, wherein: In step S1, the speckle image refers to the speckles artificially created on the sample surface, the speckles with uneven grayscale formed by the rough surface of the material, and the speckles formed on the projection image by the natural internal texture of the multi-phase material; The calculation area is a rectangular area, which is determined by setting the coordinates of the upper left corner and the lower right corner. The calculation points are evenly distributed in the calculation area.

4. The digital image correlation method based on a deep learning image matching algorithm according to claim 1, characterized in that: In step S4, a rectangular area is defined around the predetermined range by setting the length and width.

5. The digital image correlation method based on a deep learning image matching algorithm according to claim 1, characterized in that: In step S4, the method for eliminating incorrectly matched feature point pairs is as follows: Step S4.1, randomly select three groups of feature point pairs around the calculation point, calculate the affine transformation parameters, and form an initialized affine transformation model; Step S4.2: Use the initialized affine transformation model to fit all feature point pairs around the calculation point. If the calculation error of a pair of feature points is less than a certain threshold, it is considered an inlier pair, otherwise it is an outlier pair. Step S4.3: If the number of inlier pairs exceeds a certain threshold, the affine transformation model is determined to be reasonable, and all inlier pairs are used to re-estimate the model parameters to improve the model; if the number of inlier pairs is lower than the threshold, step S4.1 is restarted to eliminate incorrectly matched feature point pairs.

6. The digital image correlation method based on a deep learning image matching algorithm according to claim 5, characterized in that: In step S4, the affine transformation method is to calculate the affine transformation parameters using the coordinates of the feature points of the reference image and the coordinates of the matching feature points in the target image, and then calculate the initial value of the deformation; the first-order affine transformation equation is selected: Where (x, y) is the local coordinate of the key feature in the reference image relative to the calculation point, (x', y') is the local coordinate of the key feature in the target image relative to the calculation point, and a 11 , a 12 , a 13 , a 21 , a 22 , a 23 For affine transformation parameters, solving the six parameters requires more than three pairs of key features around the calculation point; the relationship between the affine transformation parameters and the initial deformation value is as follows: a 11 =u x +1, a 12 =u y , a 13 =u, and 21 =in x , and 22 =in y +1, and 23 =v. Where u and v represent the initial displacement of the calculation point along the x and y directions, u x ,u y Indicates the initial value of the gradient of the calculation point along the x direction displacement in the x and y directions, v x , v y Indicates the initial value of the gradient in the x and y directions when the calculation point is displaced along the y direction.

Images