Resolution robust two-dimensional digital image correlation method based on deep learning

Abstract

The invention provides a resolution robust two-dimensional digital image correlation method based on deep learning, and belongs to the technical field of digital image processing. According to the method, the deep learning network including local feature extraction, global feature capture and multi-resolution mapping is constructed, so that pixel-by-pixel displacement and strain field prediction is realized. In a training stage, a local feature extraction module extracts fine deformation features by using a convolutional neural network, and then captures global long-range dependence by using a self-attention mechanism. The multi-resolution mapping module realizes feature interpolation among different scales through a continuous feature mapping technology, and generalization of a multi-resolution scene is improved. The method does not need to manually set key parameters, is high in calculation efficiency and robustness, and is especially suitable for high-precision deformation monitoring tasks in real industrial scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a resolution-robust two-dimensional digital image correlation method based on deep learning. Background Art

[0002] Two-dimensional digital image correlation (2DDIC) is an optical mechanics technology for non-contact, full-field two-dimensional deformation measurement. This technology can perform long-distance optical imaging and tracking identification of target objects or their surface feature points without direct contact with the object, thus avoiding damage or contamination. In addition, 2DDIC has strong environmental adaptability and is suitable for a variety of complex measurement scenarios. Due to its advantages such as simple equipment, convenient operation and high measurement accuracy, 2DDIC has been widely used in experimental mechanics, biomechanics, structural health monitoring, fracture mechanics and other fields, and has become an important tool for studying the behavior of materials and structures under different loading conditions.

[0003] Traditional 2DDIC technology is mainly based on the sub-region matching method. Its core idea is to iteratively optimize the objective function based on the initially set shape function parameters to achieve the best match between the deformed image and the sub-region in the reference image. In this process, the selection of sub-region size and shape function has a decisive influence on the measurement results. Traditional methods usually require manual setting of key parameters such as sub-region size and shape function, which increases the complexity of the operation. Improper parameter selection may lead to offset, noise amplification or boundary effect of the matching area, thereby causing uncertainty and ambiguity in the measurement results. In addition, with the improvement of image resolution, the computational complexity of sub-region matching increases significantly. Especially when processing large-scale image data, the computational efficiency of the traditional DIC method is significantly reduced, and it is difficult to meet the needs of real-time measurement.

[0004] To overcome the limitations of traditional DIC technology, deep learning provides a new solution in this field. The DIC method based on deep learning does not require manual parameter setting and can automatically predict the displacement field and strain field pixel by pixel, thereby effectively avoiding information loss. However, when dealing with complex deformation scenes, deep learning methods are still limited by the diversity of training data sets and the generalization ability of the model.

[0005] At present, there have been many studies trying to apply deep learning technology to the field of 2DDIC. For example, Strain-Net uses a modified U-Net structure, which is specifically used to calculate small displacement fields, and constructs a dataset of simulated object deformation through interpolation methods to train the model; DIC-Net uses an Encoder-Decoder structure to expand the network's expression capabilities, and uses a second-order continuous deformation dataset based on Hermite elements to learn reliable displacement-strain relationships; the latest DICNet-corr reduces its dependence on real data through unsupervised learning methods.

[0006] However, these deep learning DIC methods still have several limitations. First, network training relies on deformation patterns constructed in specific data sets, resulting in poor performance when encountering unprecedented displacement patterns. Second, current networks are sensitive to image resolution, because training data sets often have fixed resolutions, making it difficult to cope with the challenges of resolution changes in practical applications. In order to address the above limitations, the present invention introduces a new deep learning method. This method can effectively integrate image information of different scales through a multi-resolution training strategy and a continuous feature mapping function, thereby ensuring stable prediction accuracy and robustness under a variety of complex scenes and resolution conditions.

[0007] In summary, the traditional 2DDIC method relies on manually set parameters and is difficult to maintain high performance when processing high-resolution images; although the existing deep learning DIC method has overcome the parameter setting problem to a certain extent, it is usually only tested on a single data set, has insufficient generalization ability, and is difficult to cope with actual complex scenes. The present invention aims to propose a resolution-robust two-dimensional digital image correlation method based on deep learning, which can realize fully automatic displacement and strain field prediction, ensure high accuracy and strong robustness under different resolutions and complex deformation scenes, and fill the gaps in the existing technology. Summary of the invention

[0008] The purpose of the present invention is to provide an image deformation measurement method based on deep learning to overcome the problems of high parameter dependence, large computational complexity, and insufficient robustness existing in traditional technologies.

[0009] To achieve the above object, the present invention provides a resolution robust two-dimensional digital image correlation method based on deep learning, comprising:

[0010] Step 1: For an object sprayed with a surface speckle pattern, a set of speckle image pairs before and after deformation are obtained through an experimental device as reference images and target images.

[0011] The images can be obtained by photographing with an industrial camera at different times, or generated by a numerical simulation method.

[0012] Step 2: Perform the required image preprocessing on the input speckle image pair.

[0013] A variety of speckle image pairs with different resolutions are generated through interpolation algorithms and normalized, and a multi-resolution training strategy is used to simulate multi-scale scenes under different camera conditions.

[0014] Step 3: Design an end-to-end neural network, whose structure is mainly divided into three modules:

[0015] Feature extraction module: A local feature extractor based on a convolutional neural network (CNN) and a residual connection network is used to extract features from the input speckle image pair and to upgrade the original image features. The Galerkin self-attention mechanism is then used to model the long-range dependency of the input image and integrate feature information across a large spatial range.

[0016] Multi-resolution mapping module: Using continuous feature mapping functions, features of different scales are remapped to a uniform resolution space to ensure the consistency of deformation field prediction.

[0017] Displacement and strain prediction module: Use multi-layer perceptron (MLP) to predict displacement field and strain field data respectively.

[0018] Specifically, the features are first input into the displacement field predictor to output the estimated displacement field, and then the displacement field and the original features of the image are cascaded into a multi-layer perceptron to output the strain field.

[0019] Step 4: Train the neural network. The specific method is as follows:

[0020] The preprocessed speckle image pairs are input into the neural network, and the displacement field and strain field prediction values ​​are obtained in sequence through the feature extraction, multi-resolution mapping and deformation field prediction modules;

[0021] The error between the network output and the true value of the deformation field is calculated by the average endpoint error (AEE);

[0022] Wherein, the average endpoint error formula is:

[0023]

[0024] Where H and W represent the height and width of the image, and u represents the displacement vector (u, v) or strain vector (∈ x ,∈ y ,∈ xy ), the subscripts p and g represent the prediction result and the ground truth, respectively.

[0025] The back-propagation algorithm is used to optimize the network parameters.

[0026] Step 5: Repeat step 4 until the error value of the neural network converges below the preset threshold, or the number of iterations reaches the preset upper limit; finally, save the model parameters as a pre-trained model file;

[0027] Step 6: Load the saved neural network parameters, take the new speckle image, perform the same preprocessing as step 2, input the image into the neural network, and output the deformation field data.

[0028] Beneficial Effects

[0029] The present invention has the following beneficial effects: the proposed resolution-robust two-dimensional digital image correlation method based on deep learning does not rely on manually set key parameters compared with the traditional 2DDIC method, and has advantages in prediction speed at large resolutions. Compared with previous deep learning methods, the present invention has better accuracy and robustness at different resolution scales and is more suitable for real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Flowchart of resolution-robust two-dimensional digital image correlation method based on deep learning

[0031] Figure 2 Schematic diagram of the resolution-robust two-dimensional digital image correlation method based on deep learning

[0032] Figure 3 Schematic diagram of feature extraction of resolution-robust two-dimensional digital image correlation method based on deep learning DETAILED DESCRIPTION

[0033] In order to more clearly illustrate the resolution-robust two-dimensional digital image correlation method based on deep learning of the present invention, the specific implementation mode of the present invention is described in detail in combination with the examples below: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation mode and a specific operation process are given, but the protection scope of the present invention is not limited to the following examples.

[0034] This embodiment proposes a resolution-robust two-dimensional digital image correlation method based on deep learning, including but not limited to five functional implementation modules: a deformation image acquisition module, a multi-resolution training module, a feature extraction module, a multi-resolution mapping module, and a deformation field prediction module.

[0035] Step 1: For an object sprayed with a surface speckle pattern, a set of speckle image pairs before and after deformation are obtained through experimental equipment as reference images and target images.

[0036] The images may be images taken by industrial cameras at different times, capturing the deformation process of real objects under the action of external forces; or they may be synthetic images generated by numerical simulation methods.

[0037] Step 2: Perform the required image preprocessing on the input speckle image pair.

[0038] Firstly, for the input speckle image pair, a variety of deformed image pairs with different resolutions are generated through interpolation algorithm, and the resolution scaling ratio is adjusted according to the preset factor, such as 1×, 0.5×, 0.25×, etc.;

[0039] Secondly, the generated multi-resolution image pairs are normalized respectively, and the pixel intensity values ​​are mapped to the interval [0,1] to unify the numerical range of the input image;

[0040] Then, a multi-resolution training strategy is adopted to randomly select image pairs of different resolutions as the input of the deep learning network during the training process to simulate multi-scale scenes under different camera conditions in actual applications. The specific steps are as follows:

[0041] 1) Let B denote the batch size and H0×W0 denote the resolution of the original image.

[0042] 2) For each batch, from a uniform distribution Randomly sample B scaling factors s in ( i ) .

[0043] 3) Then resize the image to the new resolution That is, as input data of scale H1×W1.

[0044] Step 3: Input the input speckle image pair into the deep learning model. The neural network structure is mainly divided into feature extraction module, multi-resolution mapping module and deformation field prediction module. The structure of each module is described in detail below:

[0045] Feature extraction module: Figure 3 As shown in the figure, the feature extraction module uses a convolutional neural network (CNN) and a residual connection network to extract local features, and combines the self-attention mechanism to capture global features.

[0046] Specifically, the local feature extraction part includes two separate convolutional layers and 16 residual connection blocks; the convolutional layer uses a convolution kernel of size 3×3, which is used for dimensionality reduction of input features and dimensionality increase of output features. The residual connection block contains two convolutional layers and a ReLU activation layer to ensure that the spatial dimensions of the features remain consistent. In order to retain the slight intensity changes in the speckle image, the batch normalization (BatchNorm) layer is removed from all convolutions to avoid affecting the retention of slight intensity changes in the speckle image.

[0047] like Figure 3 As shown in Figure 1, the global feature capture module uses the Galerkin self-attention mechanism, which is specifically used to capture long-range dependencies in the deformation field. Specifically, the input features are projected into query (Q), key (K), and value (V), and calculated through the Galerkin self-attention mechanism. The core expression is:

[0048]

[0049] in Ln(·) represents the layer normalization (LayerNorm) layer; It is implemented by a feedforward neural network (FFN), including two 1×1 convolutional layers and a GELU activation layer for nonlinear transformation and mapping of features; n represents the normalization factor, which is used to scale the attention weight. The Galerkin self-attention mechanism avoids Softmax calculations through function space operations and optimizes the efficiency of long-range feature extraction.

[0050] Multi-resolution mapping module: This module uses a continuous feature mapping function The extracted features are remapped to the original image resolution to ensure the reconstruction consistency of the deformation field. Let C be the feature dimension. For any spatial coordinate (x, y), the corresponding feature vector can be queried through the mapping function To achieve multi-resolution mapping, the scale parameter s is introduced and the feature mapping function is updated as: By adjusting s, the feature resolution is dynamically changed to achieve seamless conversion between different scales. The mapping function is modeled by a multi-layer perceptron (MLP), which takes spatial coordinates as input and outputs the corresponding feature vector.

[0051] At scale s, for a query point x q , the feature interpolation is defined as:

[0052]

[0053] where x l are the coordinates of the four adjacent feature points, z is their corresponding potential code, δ l (x q )=x q -x l Represents the weight calculated by bilinear interpolation. Through the continuous feature mapping method, a smooth transition between different resolutions can be achieved, and low-resolution features can be mapped to high-resolution output.

[0054] Deformation field prediction module: The deformation field prediction module includes a displacement field predictor and a strain field predictor, wherein: the displacement field predictor inputs the extracted features into the MLP and outputs the displacement tensor Where H and W represent the height and width of the image respectively, and 2 represents the horizontal displacement (u) and vertical displacement (v) components.

[0055] The strain field predictor outputs the strain tensor by concatenating the displacement field features with the original image features and then inputting them into the MLP. Corresponding to three strain components: ∈ xx ,∈ yy and ∈ xy The ability to capture deformation information is enhanced through feature sharing and fusion, thereby predicting the displacement field and strain field end-to-end.

[0056] Step 4: Train the neural network. The specific method is as follows:

[0057] First, the speckle image pair preprocessed in step 2 is input into the neural network, and the local and global features of the image are obtained through the feature extraction module.

[0058] Then, a multi-resolution mapping module is used to adjust the spatial resolution of the features to be consistent with the input by representing the image features as continuous functions to avoid feature information loss during interpolation.

[0059] Afterwards, the features are input into the displacement and strain prediction module, which outputs the predicted values ​​of displacement field and strain field respectively.

[0060] Finally, the error between the network prediction result and the ground truth is calculated by using the average endpoint error. Based on the calculated error value, the Adam optimization algorithm is used to calculate the gradient of the loss function to the neural network parameters. Through the gradient information, the network parameters are gradually adjusted to minimize the loss function value;

[0061] Step 5: Repeat step 4 until the error value of the neural network converges below the preset threshold, or the number of iterations reaches the preset upper limit; finally, save the model parameters as a pre-trained model file;

[0062] Step 6: Load the saved neural network parameters, take the new speckle image for prediction, perform the same preprocessing as step 2, and input the neural network, output the displacement field and strain field data prediction values ​​end-to-end. This implementation method can complete the whole process from inputting speckle images to deformation field prediction results in an end-to-end manner, and is suitable for efficient deformation analysis in experimental data and numerical simulation scenarios.

[0063] Through the above implementation modes, the present invention realizes a resolution-robust two-dimensional digital image correlation method based on deep learning, which effectively solves the deficiencies of traditional image correlation methods in terms of parameter dependence, computational complexity and robustness.

[0064] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A resolution-robust two-dimensional digital image correlation method based on deep learning, comprising the following steps: Step 1: For an object sprayed with a surface speckle pattern, a set of speckle image pairs before and after deformation are obtained through an experimental device as a reference image and a target image; The speckle image pairs are obtained by taking pictures at different times with an industrial camera, or generated by numerical simulation methods; Step 2: performing required image preprocessing on the input speckle image pair; Through the interpolation algorithm, a variety of speckle image pairs with different resolutions are generated and normalized, and a multi-resolution training strategy is used to simulate multi-scale scenes under different camera conditions; Step 3: Design an end-to-end neural network, whose structure is mainly divided into three modules: Feature extraction module: Convolutional neural network and residual connection network are used to extract local features, and the long-range dependency of the input image is modeled in combination with the Galerkin self-attention mechanism; Multi-resolution mapping module: uses continuous feature mapping functions to remap features of different scales to a uniform resolution space to ensure the consistency of deformation field prediction; Deformation field prediction module: Use multi-layer perceptron to predict displacement field and strain field data respectively; Step 4: Train the neural network. The specific method is as follows: The preprocessed speckle image pairs are input into the neural network, and the displacement field and strain field prediction values ​​are obtained in sequence through the feature extraction, multi-resolution mapping and deformation field prediction modules; The error between the network output result and the true value of the deformation field is calculated by averaging the endpoint error; Optimize network parameters using back-propagation algorithm; Step 5: Repeat step 4 until the error value of the neural network converges below the preset threshold, or the number of iterations reaches the preset upper limit; finally, save the model parameters as a pre-trained model file; Step 6: Load the saved neural network parameters, take a new speckle image pair, perform the same preprocessing as step 2, input the image into the neural network, and output the deformation field data.

2. The resolution-robust two-dimensional digital image correlation method based on deep learning according to claim 1, characterized in that: In step 2, the image preprocessing includes the following specific steps: Firstly, for the input speckle image pairs, a variety of speckle image pairs with different resolutions are generated through interpolation algorithm, and the resolution scaling ratio is adjusted according to the preset factor; Secondly, the generated multi-resolution speckle image pairs are normalized respectively, and the pixel intensity values ​​are mapped to the interval [0, 1] to unify the numerical range of the input image pairs; Then, a multi-resolution training strategy is adopted to simulate multi-scale scenes under different camera conditions in practical applications by randomly selecting speckle image pairs of different resolutions as the input of the deep learning network during the training process.

3. The resolution-robust two-dimensional digital image correlation method based on deep learning according to claim 1, characterized in that: In step 3, the local feature extraction adopts a combination of a convolutional neural network and a residual connection network; the local feature extraction module consists of multiple residual blocks, each of which includes two convolutional layers, a ReLU activation function and a residual path, which is used to realize layer-by-layer feature extraction; in order to retain the original pixel distribution of the input speckle image pair, the batch normalization layer is not used in the feature extraction process, thereby avoiding the normalization of the grayscale features, so as to ensure the integrity of the detail features of the input image.

4. The resolution-robust two-dimensional digital image correlation method based on deep learning according to claim 1, characterized in that: In step 3, the Galerkin self-attention mechanism converts the input features into query, key and value matrices (Q, K, V) to calculate the attention weights: in and Represents the key and value matrices after layer normalization, and n is the normalization factor of the feature dimension; the self-attention mechanism is used to model the long-range dependencies of input features, provide a global representation of the deformation field, and support unified modeling in multi-resolution scenarios.

5. The resolution-robust two-dimensional digital image correlation method based on deep learning according to claim 1, characterized in that: In step 3, the multi-resolution mapping module uses a continuous feature mapping function The extracted image features are represented as continuous spatial functions, where C represents the feature dimension. For any spatial coordinate (x, y), the corresponding feature vector can be queried through the continuous feature mapping function In order to realize multi-resolution mapping, a scale parameter s is introduced into the continuous feature mapping function, and the updated mapping form is: Where s is used to control the scaling of feature resolution; By dynamically adjusting the scale parameter s, the module can smoothly interpolate input features to adapt to scenes with different observation distances and deformation amplitudes.

6. The resolution robust two-dimensional digital image correlation method based on deep learning according to claim 1, characterized in that: In step 4, the specific method of updating the neural network parameters through the back propagation algorithm The following steps are involved: First, the error value is calculated based on the difference between the displacement field or strain field data output by the network and the ground truth. The error value is evaluated by the average endpoint error to quantify the deviation between the predicted result and the true value. The calculation formula of the average endpoint error is: Where H and W represent the height and width of the image, and u represents the displacement vector (u, v) or strain vector (∈ x ,∈ y ,∈ xy ), subscripts p and g represent the predicted value and ground truth value, respectively; Based on the calculated error value, the Adam optimization algorithm is used to calculate the gradient of the loss function to the neural network parameters. Through the gradient information, the network parameters are gradually adjusted to minimize the loss function value; By continuously iteratively optimizing the network parameters, the error value is gradually reduced to approach the difference between the network prediction results and the true value of the deformation field.

Images