Method for Measuring Displacement Field of Effective Region of Material Based on Convolutional LSTM Neural Network

Abstract

The present invention claims protection for a method for measuring the displacement field of the effective region of a material based on a convolutional LSTM neural network, comprising the following steps: Spraying random spray speckles on the surface of the material, and using a camera to continuously collect images to record the process of the material deforming under an external force. Constructing a material deformation image sequence as a data set, the data set including a training set and a test set. Combining convolution, deconvolution, multi-tasking, and a convolutional LSTM neural network to establish a neural network model that can measure the displacement field of the material in real time and segment the crack region by inputting the image sequence of the material deformation. Using the training set data to train the multi-task convolutional LSTM neural network model for material displacement field measurement and crack region segmentation. Inputting the time-series image data collected by the camera, measuring the real-time displacement field of the material and segmenting the region where cracks appear in the material, and finally obtaining the displacement field of the non-cracked effective region. The present invention can better combine spatio-temporal features to measure the deformation displacement field of the material.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence and optical measurement, and particularly relates to the technology of a displacement field measurement method for an effective area of a material. Background Art

[0002] Digital image correlation (DIC) is a full-field displacement and strain measurement technology that has been rapidly popularized in the field of experimental mechanics. It is an optical measurement method that achieves a good balance among generality, ease of use, and metrological performance. This optical measurement method was proposed in the 1980s, and in the past few decades, many scholars have improved the performance, accuracy, stability, etc. of the DIC algorithm, expanding its application scope and usability.

[0003] The traditional DIC method creates texture features by spraying random speckles on the material surface and continuously captures images with a camera to record the deformation process of the material. The effective area in the pre-deformation image is divided into grids, and each sub-region is regarded as a rigid motion. Then, for each sub-region, through a certain search method, correlation calculations are performed according to a pre-defined correlation function to find the region in the post-deformation image with the maximum cross-correlation coefficient with this sub-region, that is, the position of this sub-region after deformation, and thus the displacement of this sub-region is obtained. By calculating all sub-regions, the deformation information of the entire field can be obtained.

[0004] A digital image correlation method based on deep learning has been proposed. It inputs two continuously changing frames of images into a convolutional neural network at the same time, and through a series of convolutional and deconvolution operations, finally obtains the displacement field between the two frames of images.

[0005] Traditional convolutional neural networks can estimate the displacement field of material deformation to a certain extent, but they only target the deformation at a specific moment. The deformation of a material under the action of an external force is a continuous process, and there is a certain relationship between the deformation amount at the current moment and the past deformation. Therefore, compared with only processing the image data at a certain moment, processing the image data at more moments in combination with time series can obtain better displacement field prediction accuracy. The convolutional LSTM neural network combines the ability of the convolutional neural network to process spatial problems and the ability of LSTM to solve time series problems, and shows strong performance and theoretical advantages in solving spatio-temporal sequence prediction problems. Therefore, it is necessary to study a multi-task convolutional LSTM neural network for high-precision displacement field measurement of the effective area of a material to better solve the problems existing in traditional DIC.

[0006] After retrieval, in the closest prior art, CN108507476B, a method, device, equipment, and storage medium for measuring the displacement field on the material surface. Among them, the method includes: for the material to be measured, collect the first image before deformation and the second image after deformation, and the sizes of the first image and the second image are the same; establish the first image pyramid based on the first image, and establish the second image pyramid based on the second image; according to the first image pyramid and the second image pyramid, calculate the gray gradient of each level in the second image pyramid relative to the same level in the first image pyramid; substitute the gray gradient of each level in the second image pyramid into the objective function of the L1 norm optical flow algorithm, and use the subgradient iteration algorithm to calculate the objective function of the L1 norm optical flow algorithm to obtain the displacement field of each level in the second image pyramid. This solution realizes the calculation of the displacement field at the large deformation of the material surface and shortens the time for displacement calculation. Although this technology can handle the displacement field at large deformations well, it still belongs to the traditional displacement field measurement method and is only applicable to the case where the material does not generate cracks. When cracks occur in the material under external force, this technology cannot handle the situation where the displacement field cannot be measured due to the extension and expansion of the cracks. The present invention uses deep learning to simultaneously measure the displacement field and segment the crack area. The crack segmentation scheme based on deep learning can accurately segment the crack and non-crack areas. Since the displacement field in the crack area is meaningless, the present invention can fuse the displacement field measurement result and the crack segmentation result to achieve the purpose of only measuring the displacement field of the effective area of the material. The method in the CN108507476B patent uses the traditional method of iteratively optimizing the objective function to measure the displacement field. This method generally uses CPU resources for calculation, and its processing speed is difficult to meet the real-time requirements. The neural network constructed by the present invention can use GPU resources for acceleration and can realize real-time measurement of the displacement field on a high-performance GPU. Summary of the Invention

[0007] The present invention aims to solve the above problems of the prior art. A method for measuring the displacement field of the effective area of the material based on a convolutional LSTM neural network is proposed. The technical solution of the present invention is as follows:

[0008] A method for measuring the displacement field of the effective region of a material based on a convolutional LSTM neural network, which includes the following steps: Spraying random spray speckles on the material surface, and using a camera to continuously collect images to record the deformation process of the material under the action of an external force; Constructing a material deformation image sequence as a data set, and the material deformation image sequence data set includes a training set and a test set; Combining convolution, deconvolution, multi-task and convolutional LSTM neural networks to establish a neural network model that can measure the displacement field of the material in real time and segment the crack region by inputting the image sequence of the material deformation; Using the training set data to train the multi-task convolutional LSTM neural network model for material displacement field measurement and crack region segmentation; Using the trained material displacement field measurement and crack region segmentation model, inputting the time-series image data collected by the camera, measuring the real-time displacement field of the material and segmenting the region where cracks appear on the material, and finally obtaining the displacement field of the non-crack effective region.

[0009] Further, the step of spraying random spray speckles on the material surface and using a camera to continuously collect images to record the deformation process of the material under the action of an external force specifically includes the following limiting conditions:

[0010] The material surface is kept flat so that there are no unevenness; The speckle color has a high contrast with the material background, and black and white are used respectively; The diameter of the sprayed speckles is between 1 mm and 4 mm, and the speckles are randomly and evenly distributed on the material surface; The camera imaging plane is parallel to the material plane, that is, the camera axis line is perpendicular to the material plane. The relative position between the camera and the material remains unchanged during the image acquisition process; The distance between the camera and the material should satisfy clear imaging, and it is ensured that the imaging diameter of the speckles on the material surface should occupy 3 to 12 pixels; The image acquisition frequency of the camera remains constant, and the camera white balance and exposure time should remain constant during the acquisition process; The image sequence collected in a group of experiments should include at least 10 or more images, and there should be a crack region in the images.

[0011] Further, the step of constructing a material deformation image sequence as a data set, and the material deformation image sequence data set includes a training set and a test set, specifically includes:

[0012] Select an original image as the initial reference frame. This original image can be an image generated by an analog speckle generator, or an image from a publicly available dataset, or an image obtained through experimental acquisition. Through computer simulation of the random deformation process, perform pixel displacement operations on the original image, with continuous deformation more than 10 times. Through computer simulation of the crack generation and propagation process, divide the pixels on the image into normal regions and crack regions. Combine the first image and the second image to form the input data at the first moment, and the displacement field and crack region segmentation result of the first deformation as the true value of the network output data at the first moment. And so on, use the nth and (n + 1)th images as the input data at the nth moment, and the displacement field and crack region segmentation result of the nth computer simulation as the true value of the network output at the nth moment.

[0013] Repeat the above operations to perform simulated displacement and crack on more than 1000 images to generate a dataset including a training set and a test set.

[0014] Furthermore, combine convolutional, deconvolutional, multi-task, and convolutional LSTM neural networks to establish a neural network model for real-time measurement of the material displacement field and segmentation of the crack region by inputting an image sequence of material deformation, specifically including:

[0015] Combine convolutional, deconvolutional, convolutional LSTM neural networks, and multi-task neural networks to construct a multi-task convolutional LSTM neural network that can simultaneously measure the material displacement field and segment the crack region of the image. The convolutional LSTM layer improves the structure of the original fully connected LSTM layer, replacing the feedforward connections between gates and internal states with convolutions, achieving the simultaneous processing of temporal and spatial information. Add 4 convolutional layers before the input of the convolutional LSTM layer to perform 4 times of downsampling convolution on the input data to obtain refined features, and then use the refined features as the input of the convolutional LSTM layer. Establish two branches after the convolutional LSTM layer, which are respectively used for displacement field measurement and crack region segmentation. The displacement field measurement branch consists of 4 deconvolutional layers, with the input being the output of the convolutional LSTM layer and the output being a displacement field with the same size as the output image. The crack region segmentation branch consists of 4 deconvolutional layers, with the input being the output of the convolutional LSTM layer and the output being a semantic segmentation result with the same size as the output image. Set the non-crack regions in the semantic segmentation as valid regions, and combine with the displacement field measurement result to obtain the displacement field of the valid regions.

[0016] Furthermore, the convolutional LSTM layer improves the structure of the original fully connected LSTM layer, replacing the feedforward connections between gates and internal states with convolutions, achieving the simultaneous processing of temporal and spatial information. The specific implementation inside it is:

[0017]

[0018]

[0019]

[0020] o t = σ(W xo * X t + W ho * H t-1 + W co * H t-1 + b o )

[0021]

[0022] where i t is the input of the input gate at time t, f t is the input of the forget gate at time t, o t is the input of the output gate at time t, C t is the hidden information at time t, H t is the output information at time t, W xi 、W xf 、W xc 、W xo are the weights of X t ; W hi 、W hf 、W hc 、W ho are the weights of H t-1 ; b i 、b f 、b c 、b o are the bias factors, σ is the activation function, * represents the convolution operation, represents matrix multiplication.

[0023] Furthermore, training the multi-task convolutional LSTM neural network model for material displacement field measurement and crack region segmentation using the training set data specifically includes:

[0024] The input data format for training the multi-task convolutional LSTM neural network is [n, h, w, c], and the output displacement field and crack region segmentation data format is [n, h, w, 1], where n is the number of data frames, h is the height of the input and output images, w is the width of the input and output images, and c is the number of channels;

[0025] For the result of the displacement field measurement branch, the mean error function is used to evaluate the error between the model estimation result and the true result;

[0026]

[0027] where, (ue , v e ), representing the estimated displacements in the horizontal and vertical directions, (u g , v g ), representing the true displacements in the horizontal and vertical directions, (i, j) representing pixel coordinates, and K and L representing the regions where the AEE value is calculated.

[0028] For the result of the crack region segmentation branch, the cross-entropy function is used to evaluate the error of the model segmentation result;

[0029]

[0030] Among them, y ij is the ground truth of the segmentation result, is the segmentation result output by the model.

[0031] By weighting the two errors separately, the total error is calculated as follows:

[0032] Error = k 1 × Error 1 + k 2 × Error 2 (k 1 + k 2 = 1)

[0033] The total error is backpropagated through the chain rule, and the Adam gradient descent optimization algorithm is used to train the network: update the neural network parameters for optimization and learning to improve the accuracy of the network.

[0034] Furthermore, the training of the network using the Adam gradient descent optimization algorithm specifically includes:

[0035] First, calculate the first-order moment estimate p and the second-order moment estimate v of the gradient:

[0036]

[0037]

[0038] Among them, is the iteration number, θ is the parameter vector, E(θ) is the loss function, β 1 and β 2 respectively represent the gradient decay factors of the first-order and second-order moment estimates;

[0039] According to the calculated p and v, combined with the learning rate α and the minimum deviation ε, the updated value of θ is obtained.

[0040]

[0041] Furthermore, by using the trained material displacement field measurement and crack region segmentation model, inputting the sequential image data collected by the camera, measuring the real-time displacement field of the material and segmenting the region where cracks appear in the material, and finally obtaining the displacement field of the effective non-crack region, the specific steps are as follows:

[0042] Adjust the collected image sequence to the same format as the dataset and input it into the constructed multi-task convolutional LSTM neural network model; the input image will first undergo 4 times of convolutional downsampling for feature extraction and refinement and then be input into the convolutional LSTM layer; the output of the convolutional LSTM layer passes through the displacement field measurement branch and the crack region segmentation branch respectively to simultaneously measure the displacement field and segment the crack region; set the non-crack region as the effective region, and the displacement field of the effective region can be obtained by combining the measurement results of the displacement field; by inputting continuous deformation data, the displacement field measurement and crack region segmentation of the real-time image sequence are completed.

[0043] The advantages and beneficial effects of the present invention are as follows:

[0044] The present invention obtains the image sequence of material deformation, constructs the image sequence of material deformation as the dataset, combines convolution, deconvolution, convolutional LSTM neural network and multi-task neural network to construct a multi-task convolutional LSTM neural network that can simultaneously measure the material displacement field and segment the crack region in the image; and trains the neural network model using the data in the training set; finally, uses the trained multi-task convolutional LSTM neural network to measure the displacement field and segment the crack region of the real image sequence collected by the camera, and finally obtains the displacement field of the non-crack region. The present invention can not only accurately measure the displacement field of material deformation and segment the crack region, but also due to the characteristic that the convolutional LSTM neural network can process sequential and spatial data simultaneously, this model can improve the accuracy and efficiency of displacement field measurement and crack region segmentation.

[0045] The main innovation points of the present invention are as follows:

[0046] 1. Compared with the traditional displacement field calculation method, the present invention innovatively uses the deep learning method to realize the calculation of the displacement field, providing a brand-new calculation idea and method for material displacement field measurement. The present invention can effectively overcome the problems of narrow application range of traditional methods and inability to handle extreme situations such as cracks well.

[0047] 2. The present invention uses the method of computer simulation to generate speckles and deformation to generate the dataset for deep learning training, ensuring the accuracy and precision of the dataset and providing reliable data for training a high-precision neural network model.

[0048] 3. The essence of the process of material deformation is that the spatial characteristics change with time. The convolutional LSTM neural network used in the present invention can well combine temporal and spatial characteristics to extract and process features.

[0049] 4. The present invention uses a multi-task neural network. Two corresponding loss functions are defined for displacement field measurement and crack area segmentation to calculate the error, and the total error is obtained by weighting. Compared with the traditional method that cannot handle the problem of the crack area well, this network can simultaneously perform displacement field measurement and crack area segmentation, and further combine to obtain the displacement field of the effective area (non-crack area). BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flowchart of the implementation of the multi-task convolutional LSTM neural network for high-precision displacement field measurement of the effective area of the material provided by the present invention;

[0051] Figure 2 is a structural diagram of the implementation of a multi-task convolutional LSTM neural network for high-precision displacement field measurement of the effective area of the material shown according to an exemplary embodiment;

[0052] Figure 3 is a flowchart of using the present invention for material displacement field measurement and crack area segmentation;

[0053] Figure 4 is a schematic diagram of speckles on the surface of a material shown according to an exemplary embodiment;

[0054] Figure 5 is a schematic diagram of the process of material deformation shown according to an exemplary embodiment;

[0055] Figure 6 is a schematic diagram of the model of a multi-task convolutional LSTM neural network shown according to an exemplary embodiment;

[0056] Figure 7 is a result diagram of displacement field measurement and crack area segmentation shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] Next, the technical solutions in the embodiments of the present invention will be clearly and detailedly described in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention.

[0058] The technical solution of the present invention to solve the above technical problems is:

[0059] The flowchart of the implementation of the multi-task convolutional LSTM neural network for high-precision displacement field measurement of the effective area of the material of the present invention is as Figure 1As shown in the figure, it is specifically divided into 5 steps:

[0060] Step 1. Randomly spray speckles on the material surface, and continuously collect images using a camera to record the deformation process of the material under external forces.

[0061] Step 2. Construct an image sequence of material deformation as a data set, and the data set of the image sequence of material deformation includes a training set and a test set.

[0062] Step 3. Combine convolution, deconvolution, multi-task, and convolutional LSTM neural networks to establish a neural network model that can measure the displacement field of the material in real time and segment the crack area by inputting the image sequence of material deformation.

[0063] Step 4. Use the training set data to train the multi-task convolutional LSTM neural network model for material displacement field measurement and crack area segmentation.

[0064] Step 5. Use the trained material displacement field measurement and crack area segmentation model, input the time-series image data collected by the camera, measure the real-time displacement field of the material and segment the area where cracks appear in the material, and finally obtain the displacement field of the effective area without cracks.

[0065] The entire process can adopt a modular design, such as Figure 2 shown in the figure, including an image acquisition module, an image preprocessing module, an image data set construction module, a model construction module, a model training module, a displacement field measurement and crack area segmentation module, and an effective area displacement field measurement module.

[0066] The specific steps of obtaining the effective area displacement field and segmenting the crack area through this model are as Figure 3 shown in the figure.

[0067] As a possible implementation of this embodiment, in Step 1, randomly spray speckles on the material surface as Figure 4 shown in the figure, including the following steps:

[0068] Step 11. Since the material surface should be flat, there should be no unevenness. The color of the speckles has a high contrast with the material background, and generally black and white are used respectively.

[0069] Step 12. The diameter of the sprayed speckles should be between 1 mm and 4 mm, and the speckles should be randomly and evenly distributed on the material surface.

[0070] Step 13. The camera imaging plane is parallel to the material plane, that is, the camera axis line is perpendicular to the material plane. The relative position between the camera and the material remains unchanged during the image acquisition process.

[0071] Step 14: The distance between the camera and the material should be such that the imaging is clear, and it is ensured that the imaging diameter of the speckles on the material surface occupies 3 to 12 pixels.

[0072] Step 15: The image acquisition frequency of the camera should be kept constant, generally 5 to 20 Hz. During the acquisition process, the white balance and exposure time of the camera should be kept constant.

[0073] Step 16: The image sequence collected in a set of experiments should contain at least 10 or more images, and there should be a crack area in the images.

[0074] As a possible implementation manner of this embodiment, the images generated in the step 2 dataset are as Figure 5 shown, and include the following steps:

[0075] Step 21: Select an image as the initial reference frame. This image can be an image generated by a simulated speckle generator, an image from a publicly available dataset, or an image obtained from experimental acquisition.

[0076] Step 22: Through computer simulation of the random deformation process, perform pixel displacement operations on the original image. Deform continuously for more than 10 times.

[0077] Step 23: Through computer simulation of the crack generation and extension process, divide the pixels on the image into normal areas and crack areas.

[0078] Step 24: Combine the first image and the second image to form the input data at the first moment, and the displacement field and crack area segmentation result of the first deformation as the true value of the network output data at the first moment. And so on, use the nth and (n + 1)th images as the input data at the nth moment, and the displacement field and crack area segmentation result of the nth computer simulation as the true value of the output of the network at the nth moment.

[0079] Step 25: Repeat the above operations to perform simulated displacement and cracking on more than 1000 images to generate a dataset including a training set and a test set.

[0080] As a possible implementation manner of this embodiment, the multi-task convolutional LSTM neural network model constructed in the step 3 is as Figure 6 shown, and includes the following steps:

[0081] Step 31: Combine convolution, deconvolution, convolutional LSTM neural network and multi-task neural network to construct a multi-task convolutional LSTM neural network that can simultaneously measure the material displacement field and segment the crack area of the image.

[0082] Step 32. By improving the structure of the original fully-connected LSTM layer, the convolutional LSTM layer replaces the feedforward connections between gates and internal states with convolutions, achieving the simultaneous processing of temporal and spatial information. The specific implementation inside is as follows:

[0083]

[0084]

[0085]

[0086] o t = σ(W xo * X t + W ho * H t-1 + W co * H t-1 + b o )

[0087]

[0088] where i t is the input of the input gate at time t, f t is the input of the forget gate at time t, o t is the input of the output gate at time t, C t is the hidden information at time t, H t is the output information at time t. W xi , W xf , W xc , W xo are the weights of X t , W hi , W hf , W hc , W ho are the weights of H t-1 , b i , b f , b c , b o are the bias factors, σ is the activation function, * represents the convolution operation, represents matrix multiplication.

[0089] Step 33. Four convolutional layers are added before the input of the convolutional LSTM layer to perform 4 times of downsampling convolution on the input data to obtain refined features, and then the refined features are used as the input of the convolutional LSTM layer.

[0090] Step 34. Two branches are established after the convolutional LSTM layer, which are respectively used for displacement field measurement and crack area segmentation.

[0091] Step 35, the displacement field measurement branch consists of 4 deconvolution layers. The input is the output of the convolutional LSTM layer, and the output is a displacement field with the same size as the output image.

[0092] Step 36, the crack region segmentation branch consists of 4 deconvolution layers. The input is the output of the convolutional LSTM layer, and the output is a semantic segmentation result with the same size as the output image.

[0093] Step 37, set the non-crack regions in the semantic segmentation as valid regions. Combining with the displacement field measurement results, the displacement field of the valid regions can be obtained.

[0094] As a possible implementation of this embodiment, step 4 includes the following steps:

[0095] Step 41, the input data format for training the multi-task convolutional LSTM neural network is [n, h, w, c], and the output displacement field and crack region segmentation data format is [n, h, w, 1], where n is the number of data frames, h is the height of the input and output images, w is the width of the input and output images, and c is the number of channels.

[0096] Step 42, for the result of the displacement field measurement branch, the following average error function is used to evaluate the error between the model estimation result and the true result.

[0097]

[0098] Among them, (u e , v e ) represents the estimated displacements in the horizontal and vertical directions, (u g , v g ) represents the true displacements in the horizontal and vertical directions, (i, j) represents the pixel coordinates, and K and L represent the region where the AEE value is calculated.

[0099] Step 43, for the result of the crack region segmentation branch, the cross-entropy function is used to evaluate the error of the model segmentation result.

[0100]

[0101] Among them, y ij is the true value of the segmentation result, is the segmentation result output by the model.

[0102] Step 44, by weighting the two errors respectively, calculate the total error as follows:

[0103] Error = k 1 × Error 1 + k 2 × Error 2 (k1 +k 2 = 1)

[0104] Step 45, backpropagate the total error through the chain rule and train the network using the Adam gradient descent optimization algorithm:

[0105] First, calculate the first-order moment estimate p and the second-order moment estimate v of the gradient:

[0106]

[0107]

[0108] where is the number of iterations, θ is the parameter vector, E(θ) is the loss function, and β 1 and β 2 represent the gradient decay factors of the first-order and second-order moment estimates respectively.

[0109] Based on the calculated p and v, combined with the learning rate α and the minimum deviation ε, obtain the updated value of θ.

[0110]

[0111] Use the updated θ to optimize and learn the neural network parameters to improve the accuracy of the network.

[0112] As a possible implementation of this embodiment, the model output result of step 5 is as follows Figure 7 shown and includes the following steps:

[0113] Step 51, adjust the image sequence collected in step 2 to the same format as the dataset and input it into the constructed multi-task convolutional LSTM neural network model.

[0114] Step 52, the input image will first be subjected to 4 times of convolutional downsampling for feature extraction and refinement and then input into the convolutional LSTM layer.

[0115] Step 53, the output of the convolutional LSTM layer passes through the displacement field measurement branch and the crack region segmentation branch respectively to simultaneously measure the displacement field and segment the crack region.

[0116] Step 54, set the non-crack region as the valid region, and the displacement field of the valid region can be obtained by combining the measurement results of the displacement field.

[0117] Step 55, by inputting continuous deformation data, complete the displacement field measurement and crack region segmentation of the real-time image sequence.

[0118] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0119] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0121] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A method for measuring the displacement field of the effective region of a material based on a convolutional LSTM neural network, characterized in that, it includes the following steps: Spray random speckles on the surface of the material, and use a camera to continuously collect images to record the deformation process of the material under the action of external forces; Construct a material deformation image sequence as a data set. The material deformation image sequence data set includes a training set and a test set; Combine convolution, deconvolution, multi-task and convolutional LSTM neural networks to establish a neural network model that can measure the displacement field of the material in real time and segment the crack area by inputting the image sequence of the material deformation; Use the training set data to train the multi-task convolutional LSTM neural network model for material displacement field measurement and crack area segmentation; Use the trained material displacement field measurement and crack area segmentation model, input the time-series image data collected by the camera, measure the real-time displacement field of the material and segment the area where cracks appear in the material, and finally obtain the displacement field of the non-crack effective area; The combination of convolution, deconvolution, multi-task and convolutional LSTM neural networks to establish a neural network model that can measure the displacement field of the material in real time and segment the crack area by inputting the image sequence of the material deformation specifically includes: Combine convolution, deconvolution, convolutional LSTM neural network and multi-task neural network to construct a multi-task convolutional LSTM neural network that can simultaneously measure the displacement field of the material and segment the crack area of the image; The convolutional LSTM layer improves the structure of the original fully connected LSTM layer, replaces the feed-forward connections between gates and internal states with convolution, and realizes the simultaneous processing of time and space information; Add 4 convolutional layers before the input of the convolutional LSTM layer, perform 4 times of downsampling convolution on the input data to obtain refined features, and then use the refined features as the input of the convolutional LSTM layer; Establish two branches after the convolutional LSTM layer, which are used for displacement field measurement and crack area segmentation respectively; The displacement field measurement branch consists of 4 deconvolution layers, the input is the output of the convolutional LSTM layer, and the output is a displacement field with the same size as the output image; The crack area segmentation branch consists of 4 deconvolution layers, the input is the output of the convolutional LSTM layer, and the output is a semantic segmentation result with the same size as the output image; Set the non-crack area in the semantic segmentation as the effective area, and combine the displacement field measurement result to obtain the displacement field of the effective area.

2. The method for measuring the displacement field of the effective region of a material based on a convolutional LSTM neural network according to claim 1, characterized in that, the spraying of random speckles on the surface of the material and using a camera to continuously collect images to record the deformation process of the material under the action of external forces specifically includes the following limiting conditions: The surface of the material remains flat, such that there are no unevennesses; the speckle color has a high contrast with the material background, being black and white respectively; the diameter of the sprayed speckles is between 1 mm and 4 mm, and the speckles are randomly and evenly distributed on the material surface; the camera imaging plane is parallel to the material plane, i.e., the camera axis line is perpendicular to the material plane, and the relative position between the camera and the material remains unchanged during the image acquisition process; the distance between the camera and the material should satisfy clear imaging, and it is ensured that the imaging diameter of the speckles on the material surface should occupy 3 to 12 pixels; the image acquisition frequency of the camera remains constant, and the camera white balance and exposure time should remain constant during the acquisition process; the image sequence collected in a group of experiments should contain at least 10 or more images, and there are crack regions in the images.

3. A method for measuring the displacement field of an effective region of a material based on a convolutional LSTM neural network according to claim 1, characterized in that constructing a material deformation image sequence as a data set, the material deformation image sequence data set including a training set and a test set, specifically including: Selecting an original image as an initial reference frame, which is an image generated by a simulated speckle generator, or an image from a publicly available data set, or an image obtained by experimental acquisition; simulating a random deformation process by computer, performing pixel point displacement operations on the original image, and continuously deforming it more than 10 times; simulating the generation and extension process of cracks by computer, and dividing the pixels on the image into normal regions and crack regions; forming the first image and the second image into the input data at the first moment, and the displacement field and crack region segmentation result of the first deformation as the true value of the network output data at the first moment; and so on, taking the nth and the n + 1th images as the input data at the nth moment, and the displacement field and crack region segmentation result of the nth computer simulation as the true value of the network output at the nth moment; Repeating the above operations, simulating displacements and cracks for more than 1000 images to generate a data set including a training set and a test set.

4. A method for measuring the displacement field of an effective region of a material based on a convolutional LSTM neural network according to claim 3, characterized in that the convolutional LSTM layer improves the structure of the original fully connected LSTM layer, replacing the feed-forward connections between gates and between internal states with convolutions, realizing the simultaneous processing of temporal and spatial information, and its specific internal implementation is: o t = σ(W xo * X t + W ho * H t-1 + W co * H t-1 + b o ) where, i t is the input of the input gate at time t, f t is the input of the forget gate at time t, o t is the input of the output gate at time t, C t is the hidden information at time t, H t is the output information at time t, W xi 、W xf 、W xc 、W xo are the weights of X t , W hi 、W hf 、W hc 、W ho are the weights of H t-1 , b i 、b f 、b c 、b o are the bias factors, σ is the activation function, * represents the convolution operation, represents matrix multiplication.

5. A method for measuring the displacement field of an effective region of a material based on a convolutional LSTM neural network according to claim 4, characterized in that using the training set data to train a multi-task convolutional LSTM neural network model for material displacement field measurement and crack region segmentation, specifically including: The input data format for training the multi-task convolutional LSTM neural network is [n, h, w, c], and the output displacement field and crack region segmentation data format is [n, h, w, 1], where n is the number of data frames, h is the height of the input and output images, w is the width of the input and output images, and c is the number of channels; For the results of the displacement field measurement branch, the mean error function is used to evaluate the error between the model estimation results and the true results; where (u e , v e ) represents the estimated displacements in the horizontal and vertical directions, (u g , v g ) represents the true displacements in the horizontal and vertical directions, (i, j) represents the pixel coordinates, and K and L represent the regions where the AEE values are calculated; For the results of the crack region segmentation branch, the cross-entropy function is used to evaluate the error of the model segmentation results; Among them, y ij is the true value of the segmentation result, and is the segmentation result output by the model; By weighting the two errors separately, the total error is calculated as follows: Error = k 1 × Error 1 + k 2 × Error 2 (k 1 + k 2 = 1) The total error is backpropagated through the chain rule, and the network is trained using the Adam gradient descent optimization algorithm: update the neural network parameters for optimization and learning to improve the accuracy of the network.

6. A method for measuring the displacement field of the effective region of a material based on a convolutional LSTM neural network according to claim 5, wherein, the training of the network using the Adam gradient descent optimization algorithm specifically includes: First, calculate the first-order moment estimate p and the second-order moment estimate v of the gradient: where, is the number of iterations, θ is the parameter vector, E(θ) is the loss function, β 1 and β 2 represent the gradient decay factors for the first-order and second-order moment estimates, respectively; According to the calculated p and v, combined with the learning rate α and the minimum deviation ε, the updated value θ is obtained, 7. A method for measuring the displacement field of the effective region of a material based on a convolutional LSTM neural network according to claim 6, wherein, using the trained material displacement field measurement and crack region segmentation models, input the sequential image data collected by the camera, measure the real-time displacement field of the material and segment the region where cracks appear in the material, and finally obtain the displacement field of the non-crack effective region, specifically including: Adjust the collected image sequence to the same format as the dataset and input it into the constructed multi-task convolutional LSTM neural network model; the input image will first undergo 4 times of convolutional downsampling for feature extraction and refinement and then be input into the convolutional LSTM layer; the outputs of the convolutional LSTM layer pass through the displacement field measurement branch and the crack region segmentation branch respectively to simultaneously measure the displacement field and segment the crack region; set the non-crack region as the effective region, and the displacement field of the effective region can be obtained by combining the measurement results of the displacement field; by inputting continuous deformation data, the displacement field measurement and crack region segmentation of the real-time image sequence are completed.

Images