DIC speckle image generation method based on deep learning neural network

Abstract

The invention discloses a DIC speckle image generation method based on a deep learning neural network, and the method comprises the steps: taking an actual surface image of an object as a content image, and taking a speckle image as a style image; firstly, an input image is segmented into a plurality of small blocks serving as minimum calculation units, then input vector dimensions are converted into a preset value C through a linear embedding layer, subsequent processing is facilitated, and then multi-head self-attention calculation is conducted in divided windows. Small blocks are merged through down-sampling operation, resolution is reduced, the number of channels is adjusted to form hierarchical design, meanwhile, certain calculation amount is reduced, and after merging is completed, multi-head self-attention calculation continues to be conducted in a window. And then repeating the operation of the previous step, further combining the minimum calculation unit, and performing multi-head self-attention calculation in the window by using a new calculation unit. And then through a three-layer transmission module, the speckle features of the style image are migrated to the content image after decoding by a decoder, so that a speckle image containing the surface features of the object is generated. According to the method, the generated speckle picture can be ensured to have high quality of an original input speckle pattern picture, the characteristics of the actual surface of an object can be reserved, high-quality speckles adaptive to the surface of the object are obtained, and the influence of human factors on preparation of the speckles is eliminated, so that the precision when a DIC method is applied to perform stress strain measurement is ensured.

Description

Technical Field

[0001] The present invention relates to the field of digital image correlation (DIC) and image style migration, and in particular to a method for generating a DIC speckle image based on a deep learning neural network. Background Art

[0002] Image style transfer is an important branch in the field of computer vision. It aims to transfer the style of a specified style image to another image or style, thereby generating a content image or video with that style. Image style transfer mainly relies on deep learning neural networks, especially VGG networks. It extracts features from content images and style images through trained network models, and generates transfer images while minimizing style loss and content loss. It has the advantages of strong versatility and good applicability.

[0003] Digital image correlation technology is a non-contact optical measurement method that can be used for stress and strain measurement. It has many advantages, such as good full-field measurement capability, simple optical path structure, strong operability, strong environmental adaptability and non-contact measurement. An important part of DIC measurement is to spray speckle on the surface of the object. By matching the position of the same pixel point (with the same grayscale value) before and after deformation, the displacement vector of the point is obtained, thereby obtaining the displacement map of the whole field. The accuracy of pixel matching before and after deformation determines the accuracy of DIC measurement. Pixel matching uses grayscale value as a carrier, and grayscale value is generated by the sprayed speckle. Therefore, the quality of speckle greatly affects the accuracy of DIC measurement, and high-quality speckle is of great help in improving the accuracy of DIC measurement. However, artificially sprayed speckle has great uncertainty, and how to stably obtain high-quality speckle has become a problem to be solved. Summary of the invention

[0004] In order to solve the shortcomings of the above-mentioned technologies, the present invention applies image style transfer technology to DIC speckle production, aiming to stably obtain high-quality speckles for different surfaces.

[0005] To achieve the above object, the present invention proposes a DIC speckle image generation method based on a deep learning neural network, comprising the following steps:

[0006] Step 1: Input the actual surface image of the object as the content image and the high-quality speckle pattern image as the style image.

[0007] Step 2: Divide the image into many small blocks as the minimum computing unit, and then transform the input vector dimension into the preset value C through the linear embedding layer.

[0008] The image is input in RGB format with a size of H×W×3.

[0009] Among them, the linear embedding layer linearly transforms the input vector dimension and converts the vector dimension into a preset value C.

[0010] Step 3: Perform multi-head self-attention calculation within the set window.

[0011] The S2WAT model is used for multi-head self-attention calculation. Specifically, the image tensor is first filled with 0 to avoid loss of image boundary information, and then the SpW attention module is used to perform multi-head self-attention calculation. After the calculation is completed, the filling is cancelled.

[0012] Step 4: Merge the small blocks in step 2 through downsampling operation, reduce the resolution, adjust the number of channels to form a hierarchical design, and continue to perform multi-head self-attention calculations within the window after the merger is completed.

[0013] Among them, when merging, the four units separated by a minimum computing unit in the two directions are spliced, and then concat spliced ​​in the depth direction, through a LayerNorm layer, and then through a fully connected layer to make linear changes, to obtain a tensor with half the size and double the number of channels.

[0014] Among them, the multi-head self-attention calculation process in step 4 is the same as the multi-head self-attention calculation process in step 3.

[0015] Step 5: Repeat step 4 to further merge the minimum computing units and use the new computing units to perform multi-head self-attention calculations within the window.

[0016] Step 6: Transport layer transmission.

[0017] See also Figure 3 , where the transmission module is a three-layer hierarchical structure, the initial attention module is MSA, the second attention module is MHA, and the third layer adopts MLP module. Each of these three modules is connected with a LayerNorm module.

[0018] Step 7: The decoder decodes and outputs the image.

[0019] Among them, the decoder uses mirrored VGG to decode stylized features.

[0020] The computing model used in the above process is trained through the S2WAT pre-training model.

[0021] The present invention discloses a DIC speckle image generation method based on a deep learning neural network, which includes inputting an actual surface image of an object as a content image and a high-quality speckle pattern image as a style image; dividing the image into a plurality of small blocks as minimum computing units, and then transforming the input vector dimension into a preset value C through a linear embedding layer; performing multi-head self-attention calculation within a set window; merging the small blocks in step 2 through a downsampling operation, reducing the resolution, and adjusting the number of channels to form a hierarchical design, and continuing to perform multi-head self-attention calculation within the window after the merging is completed; repeating step 4, further merging the minimum computing units and using the new computing units to perform multi-head self-attention calculation within the window; transmitting through a transmission layer; and decoding and outputting the image by a decoder.

[0022] The advantage of the present invention is that, different from the randomness of artificial spraying speckle and other spraying methods, the present invention adaptively generates high-quality speckle suitable for the surface according to the actual surface conditions of the object, has the advantages of strong adaptability and good versatility, greatly reduces the influence of human factors, thereby improving the DIC measurement accuracy and ensuring good experimental operability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of DIC speckle image generation based on a deep learning neural network according to the present invention.

[0024] Figure 2 This is a model structure diagram of the present invention.

[0025] Figure 3 This is a transmission module diagram provided by the present invention.

[0026] Figure 4 This is an example diagram of speckle generation provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the principles, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be emphasized that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0028] Figure 1 For a flowchart of DIC speckle image generation based on deep learning neural network of the present invention, please refer to Figure 1 The present invention proposes a DIC speckle image generation method based on a deep learning neural network, comprising the following steps:

[0029] Step 1: Input the actual surface image of the object as the content image and the high-quality speckle pattern image as the style image.

[0030] Figure 2 For the model structure diagram of the present invention, refer to Figure 2 ,Specifically, the actual surface image of the object is input as the ,content image Ic, and the high quality speckle pattern image is input as the ,style image Is.

[0031] Step 2: Divide the image into many small blocks as the minimum computing unit, and then transform the input vector dimension into the preset value C through the linear embedding layer.

[0032] Specifically, the size of each small block as the minimum computing unit is 2×2×3, where 2×2 represents the unit size and 3 represents the dimension of the unit.

[0033] See also Figure 2 Specifically, when the image is input in RGB format and the size is H×W×3, the number of windows is set to 4. After segmentation, the image size becomes

[0034] Among them, the linear embedding layer linearly transforms the input vector dimension, changing the vector dimension from 12 to the preset value C.

[0035] Specifically, after the linear embedding layer, the size of the image becomes

[0036] Step 3: Perform multi-head self-attention calculation within the set window.

[0037] The S2WAT model is used for multi-head self-attention calculation. Specifically, the image tensor is first filled with 0 to avoid loss of image boundary information, and then the SpW attention module is used to perform multi-head self-attention calculation. After the calculation is completed, the filling is cancelled.

[0038] Step 4: Merge the small blocks in step 2 through downsampling operation, reduce the resolution, adjust the number of channels to form a hierarchical design, and continue to perform multi-head self-attention calculations within the window after the merger is completed.

[0039] Among them, when merging, the four units separated by a minimum computing unit in the two directions are spliced, and then concat spliced ​​in the depth direction, through a LayerNorm layer, and then through a fully connected layer to make linear changes, to obtain a tensor with half the size and double the number of channels.

[0040] The process of performing multi-head self-attention calculation within the window is the same as in step 3. First, tensor padding is performed, and then the SpW attention module is used to perform multi-head self-attention calculation. After the calculation is completed, the padding is canceled.

[0041] See also Figure 2 , the size of the image becomes

[0042] Step 5: Repeat step 4 to further merge the minimum computing units and use the new computing units to perform multi-head self-attention calculations within the window.

[0043] Specifically, the merging step is performed first. The merging principle is the same as in step 4, and a tensor with a size halved and a number of channels doubled is obtained.

[0044] After the merging is completed, multi-head self-attention calculation is performed. The process is the same as in steps 3 and 4. First, tensor filling is performed, and then the SpW attention module is used to perform multi-head self-attention calculation. After the calculation is completed, the filling is canceled.

[0045] See also Figure 2 , the size of the image becomes

[0046] Step 6: Transport layer transmission.

[0047] Figure 3 The transmission module diagram provided by the present invention, specifically, the transmission module is a three-layer hierarchical structure, the initial attention module is MSA, the second layer attention module is MHA, the third layer adopts MLP module, and each of these three modules is connected with a LayerNorm module.

[0048] Step 7: The decoder decodes and outputs the image.

[0049] Specifically, the decoder uses mirrored VGG to decode stylized features.

[0050] See also Figure 2 , the output speckle image containing the surface features of the object is Ics.

[0051] Figure 4 For an example diagram of speckle generation provided by the present invention, see Figure 4 When the input speckle pattern has a high quality, the generated speckle image can also maintain a high quality, with an appropriate speckle size, an appropriate speckle depth and a relatively uniform speckle distribution density.

[0052] For details, see Figure 4 , where Figure a, Figure b, and Figure c are input speckle pattern images. It can be clearly seen that speckle image a has the highest speckle quality. Figure d, Figure e, and Figure f are input content images. Figure d1, Figure d2, and Figure d3 are speckle images generated with Figure a, Figure b, and Figure c as style images and Figure d as content image. Figure e1, Figure e2, and Figure e3 are speckle images generated with Figure a, Figure b, and Figure c as style images and Figure e as content image. Figure f1, Figure f2, and Figure f3 are speckle images generated with Figure a, Figure b, and Figure c as style images and Figure f as content image.

[0053] Specifically, it can be seen that the speckle quality of Figure d1 is higher than that of Figure d2 and Figure d3, the speckle quality of Figure e1 is higher than that of Figure e2 and Figure e3, and the speckle quality of Figure f1 is higher than that of Figure f2 and Figure f3. This shows that when the input speckle pattern has a high quality, the generated speckle image can also maintain a high quality. Therefore, this method can be used to obtain speckle images for different object surfaces.

[0054] The present invention discloses a DIC speckle image generation method based on a deep learning neural network, which includes inputting an actual surface image of an object as a content image and a high-quality speckle pattern image as a style image; dividing the image into a plurality of small blocks as minimum computing units, and then transforming the input vector dimension into a preset value C through a linear embedding layer; performing multi-head self-attention calculation within a set window; merging the small blocks in step 2 through a downsampling operation, reducing the resolution, and adjusting the number of channels to form a hierarchical design, and continuing to perform multi-head self-attention calculation within the window after the merging is completed; repeating step 4, further merging the minimum computing units and using the new computing units to perform multi-head self-attention calculation within the window; transmitting through a transmission layer; and decoding and outputting the image by a decoder.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the principles and principles of the present invention that can be easily thought of by any technician familiar with the technical field should be included in the protection scope of the present invention.

Claims

1. A DIC speckle image generation method based on deep learning neural network, characterized in that: The steps include: Step 1: Input the actual surface image of the object as the content image and the high-quality speckle pattern image as the style image; Step 2: Divide the image into many small blocks as the minimum computing unit, and then transform the input vector dimension into the preset value C through the linear embedding layer; Step 3: Perform multi-head self-attention calculation within the set window; Step 4: Merge the small blocks in step 2 through downsampling operation, reduce the resolution, adjust the number of channels to form a hierarchical design, and continue to perform multi-head self-attention calculation in the window after the merging is completed; Step 5: Repeat step 4 to further merge the minimum computing units and use the new computing units to perform multi-head self-attention calculations within the window; Step 6: Transport layer transmission; Step 7: The decoder decodes and outputs the image.

2. A method for generating a DIC speckle image based on a deep learning neural network as claimed in claim 1, characterized in that: Input the actual surface image of the object as the content image and the high-quality speckle pattern image as the style image.

3. A method for generating a DIC speckle image based on a deep learning neural network as claimed in claim 1, characterized in that: A unit merging operation is required through downsampling between each two window multi-head self-attention calculations.

4. A method for generating a DIC speckle image based on a deep learning neural network as claimed in claim 1, characterized in that: The transmission module is a three-layer hierarchical structure. The initial attention module is MSA, the second attention module is MHA, and the third layer adopts MLP module. Each of these three modules is connected with a LayerNorm module.

5. A method for generating a DIC speckle image based on a deep learning neural network as claimed in claim 1, characterized in that: The decoder uses mirrored VGG to decode stylized features.

Images