Microstructure reconstruction method and system based on deep learning of rock images

Abstract

The present application provides a method and system for reconstructing microstructures based on deep learning of rock images, which relates to the field of image processing technology. The method includes: obtaining an image of the rock to be reconstructed; dividing the image into blocks and mapping it to a space of specified dimensions to obtain multiple subspaces, and obtaining global features based on the long-range dependencies between the multiple subspaces; extracting grain boundaries and microcracks in the image to obtain local features, where the grain boundary is the boundary of the crystal in the image, the crystal is a component of the rock to be reconstructed, and the microcrack is the gap between two adjacent crystals; fusing global features and local features to obtain a multi-scale joint embedding vector; reconstructing the outline of the grain boundary according to the multi-scale joint embedding vector to obtain a reconstructed image. When reconstructing the image of the rock to be reconstructed, the present application focuses on the information at multiple scales of the image to improve the accuracy of the reconstructed image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for reconstructing microstructures based on deep learning of rock images. Background Art

[0002] The mesostructure of rocks significantly influences their macroscopic mechanical behavior. Currently, research on rock mesostructure relies primarily on physical experiments and empirical models based on statistical characteristics. Physical experiments, such as those using electronic devices like computed tomography (CT) and scanning electron microscopes (SEM), scan the rock mesostructure and conduct corresponding experiments based on the scanned data. Empirical models primarily utilize deep learning techniques, which are increasingly being used in the reconstruction of rock mesostructures with the advancement of deep learning technology.

[0003] However, the above research methods still have significant technical bottlenecks: first, deep learning technology usually uses convolutional neural networks (CNN). In the feature extraction process, the empirical model based on CNN lacks the technical means to take into account the global correlation characteristics of mineral distribution and the local detail representation of grain boundary microstructure, resulting in accuracy loss in the reconstruction results at both macro and micro scales; second, the existing methods have not yet established an end-to-end mapping framework between rock mesostructure image data and key physical parameters (such as mineral content, grain boundary roughness, etc.), resulting in the inability to achieve coordinated evolution of the mesostructure reconstruction process and physical property optimization, and the accuracy of mesostructure reconstruction cannot be guaranteed.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] To address the above issues, this application provides a method and system for reconstructing microstructures based on deep learning of rock images. By fusing information at multiple scales of the image, the accuracy of the reconstruction results is guaranteed. The technical solutions of this application are as follows:

[0006] First, a method for reconstructing microstructures based on deep learning of rock images is provided, including:

[0007] Acquire an image of the rock to be reconstructed;

[0008] After dividing the image into blocks and mapping them into a space of a specified dimension, a plurality of subspaces are obtained, and a global feature is obtained based on a long-range dependency relationship between the plurality of subspaces;

[0009] Extracting grain boundaries and microcracks in the image to obtain local features, wherein the grain boundaries are boundaries of crystals in the image, the crystals are components of the rock to be reconstructed, and the microcracks are gaps between two adjacent crystals;

[0010] fusing the global features and the local features to obtain a multi-scale joint embedding vector;

[0011] The contour of the grain boundary is reconstructed according to the multi-scale joint embedding vector to obtain a reconstructed image.

[0012] In a possible implementation, reconstructing the outline of the grain boundary according to the multi-scale joint embedding vector to obtain a reconstructed image includes:

[0013] Obtaining a contour offset of the grain boundary;

[0014] Adjusting the multi-scale joint embedding vector according to the contour offset to obtain an input image;

[0015] The reconstructed image is obtained according to the input image and a standard image, wherein the standard image refers to an image showing the mineral composition, crystal arrangement, and pore distribution of the rock to be reconstructed.

[0016] In a possible implementation, adjusting the multi-scale joint embedding vector according to the contour offset to obtain the input image includes:

[0017] ,

[0018] in, is the input image, is the multi-scale joint embedding vector, is the image of the rock to be reconstructed, is the preset weight corresponding to the multi-scale joint embedding vector, F( ) is a deformable convolution function, and the deformable convolution function is used to adjust the multi-scale joint embedding vector according to the contour offset to obtain the input image, is the outline offset.

[0019] In a possible implementation, obtaining the reconstructed image according to the input image and the standard image includes:

[0020] Using a generator to obtain at least one candidate image according to the input image;

[0021] Using a discriminator to obtain the similarity between the candidate image and the standard image based on the candidate image;

[0022] The candidate image with the highest similarity is selected as the reconstructed image.

[0023] In a possible implementation, after obtaining the image of the rock to be reconstructed, the method further includes:

[0024] Using a Gaussian filtering algorithm to filter out interference signals in the image;

[0025] A histogram equalization method is used to enhance the contrast of the image.

[0026] In a second aspect, a system for reconstructing a microstructure based on deep learning of rock images is provided. The system is used to execute the above-mentioned method for reconstructing a microstructure based on deep learning of rock images, comprising:

[0027] A data acquisition layer, used to obtain images of rocks to be reconstructed;

[0028] a data extraction layer, comprising a first extraction sublayer and a second extraction sublayer arranged in parallel, wherein the first extraction sublayer is used to extract global features of the image, and the second extraction sublayer is used to extract local features of the image;

[0029] a data processing layer, connected to the first extraction sublayer and the second extraction sublayer, respectively, for fusing the global features and the local features to obtain the multi-scale joint embedding vector;

[0030] A data generation layer is used to obtain a reconstructed image according to the multi-scale joint embedding vector.

[0031] In a possible implementation, the first extraction sublayer uses a ViT model to extract global features of the image, and the ViT model is used to perform the following operations:

[0032] Linear embedding labeling: the image is divided into blocks and mapped to a space of specified dimensions to obtain multiple subspaces;

[0033] Position encoding: After capturing the spatial order of multiple subspaces, a spatial sequence is generated;

[0034] Weight adjustment: adjusting the weight of each subspace according to the long-range dependencies between the multiple subspaces in the spatial sequence;

[0035] Extract global features.

[0036] In a possible implementation, the second extraction sublayer uses a MobileNetV3 model to extract local features of the image;

[0037] The MobileNetV3 model includes at least a separable convolutional network, a nonlinear activation network and a residual network.

[0038] In a possible implementation, the data processing layer uses a cross-modal attention mechanism to fuse the global features and the local features;

[0039] The calculation formula of the cross-modal attention mechanism is:

[0040] ,

[0041] Among them, G represents the global feature, L represents the local feature, and both G and L are dimensional vector, A is the multi-scale joint embedding vector, is the function of the cross-modal attention mechanism.

[0042] In a possible implementation, the system further includes a model training layer, wherein the model training layer is used to train a reconstruction model, wherein the reconstruction model is obtained by combining the data acquisition layer, the data extraction layer, the data processing layer, and the data generation layer;

[0043] The model training layer includes:

[0044] A first training sublayer is used to construct a loss function, wherein the loss function is used to constrain the reconstruction model;

[0045] The second training sublayer uses a five-fold cross-validation method to evaluate the accuracy of the reconstructed image output by the reconstruction model;

[0046] The third training sublayer: adjusts the parameters of the reconstruction model and selects the best set of parameters as the parameters of the reconstruction model.

[0047] The technical solutions provided in the embodiments of this application can achieve the following technical effects.

[0048] (1) This application extracts global and local features from an image, then fuses them to obtain data involving information at multiple scales, i.e., a multi-scale joint embedding vector. The resulting multi-scale joint embedding vector is then adjusted to generate a reconstructed image. This indicates that when reconstructing an image, this application focuses on information at multiple scales, thereby improving the accuracy of the resulting reconstructed image.

[0049] (2) This application also provides a reconstruction model, which consists of multiple layers, each layer of which contains multiple network models. Through the mutual cooperation between the multiple network models, technical support is provided for focusing on and integrating information at multiple scales of the image, thereby ensuring the accuracy of the obtained reconstructed image.

[0050] (3) This application also provides a model training layer, which trains and optimizes the reconstruction model to ensure the accuracy of the reconstructed image obtained by the reconstruction model when reconstructing the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0052] Figure 1 This is a flow chart of the microstructure reconstruction method based on deep learning of rock images provided in an embodiment of the present application.

[0053] Figure 2 This is a structural diagram of a microstructure reconstruction system based on deep learning of rock images provided in an embodiment of the present application.

[0054] Figure 3 It is a structural diagram of the data extraction layer and data processing layer in the system embodiment of the present application.

[0055] Figure 4 This is a structural diagram of the first extraction sublayer in the system embodiment of the present application.

[0056] Figure 5 This is a flowchart of establishing a training set in an embodiment of the system of this application.

[0057] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0059] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that such usage is interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the term "including" and its variations are to be interpreted as open-ended terms meaning "including but not limited to."

[0060] like Figure 1 As shown, the present application provides a method for reconstructing a microstructure based on deep learning of rock images, which mainly includes the following steps S101 to S105.

[0061] Step S101: Acquire an image of the rock to be reconstructed.

[0062] The rock to be reconstructed refers to the rock whose image is to be reconstructed, and the image of the rock to be reconstructed refers to the image obtained by scanning the rock to be reconstructed using electronic equipment such as CT or SEM. The image not only shows the components of the rock to be reconstructed - crystals, but also the boundaries of the crystals - grain boundaries, and even the gaps between crystals - microcracks.

[0063] After obtaining the image, the image is preprocessed, including:

[0064] (1) Denoising: Use Gaussian filtering algorithm to filter out interference signals in the image. The specific calculation formula is:

[0065] ,

[0066] in, is the pixel coordinate of the image, is the Gaussian kernel function, , is the Gaussian kernel standard deviation, which is used to control the filtering strength. Determine the Gaussian kernel size by adjusting and The value of can remove noise interference in the image while retaining the microstructure of the image. The microstructure mainly refers to the above-mentioned crystals, grain boundaries, microcracks, and the shape and size of each structure.

[0067] (2) Enhancement processing: Using the histogram equalization method, by adjusting the grayscale histogram of the image, the dynamic range of the image grayscale value is expanded, the contrast of the image is enhanced, and the crystal and grain boundary structures in the image are clearer. In a specific example, the initial grayscale value range of the image is [a, b]. Indicates that the grayscale value range of the image after histogram equalization is expanded to [0,255], then we get:

[0068] ,

[0069] in, is the image after expanding the grayscale value of the image, The initial grayscale value of the image is The number of pixels, is the cumulative grayscale histogram.

[0070] After the image is pre-processed, the process proceeds to steps S102 and S103 to further process the image.

[0071] In step S102 , the image is divided into blocks and mapped into a space of a specified dimension to obtain multiple subspaces, and global features are obtained based on the long-range dependencies between the multiple subspaces.

[0072] In this embodiment, the image can be divided into multiple blocks of uniform size, or multiple blocks of unequal size, as long as the resulting blocks do not overlap. When dividing, a tiled partitioning method can be used, that is, each area on the image is divided sequentially according to a certain order. Of course, a classified partitioning method can also be used, that is, different types of areas on the image are distinguished according to their type or attributes. For example, the area where a crystal is located is divided into one block, the area where the grain boundary between two adjacent crystals is located is divided into another block, and other blank areas are treated as another separate block.

[0073] Based on the resulting multiple blocks, each block is first converted into a vector, and then the vector is linearly projected onto a space of a specified dimension to obtain a subspace corresponding to each block. Next, the spatial position of each subspace in the actual scene is analyzed, and the spatial order of each subspace is determined based on this position information. The multiple subspaces are then output as a spatial sequence based on this spatial order, for example, outputting multiple subspaces as a spatial sequence in an inner-outer or top-down order.

[0074] Based on the generated spatial sequence, this spatial sequence is used as the current spatial sequence. Multiple other spatial sequences before and after the current spatial sequence are retrieved. These retrieved spatial sequences and the current spatial sequence belong to the same rock to be reconstructed. The dependencies between the current spatial sequence and the multiple spatial sequences before and after it are then compared. For example, if the crystals of the rock to be reconstructed change after the current spatial sequence, the crystals of the rock to be reconstructed in the other spatial sequences after the current spatial sequence retain the same changes. Therefore, by comparing the current spatial sequence with the multiple spatial sequences before and after it, the dependencies between multiple subspaces within a certain time range can be determined. These dependencies are also called long-range dependencies. Finally, a self-attention mechanism is used to adjust the weights of each subspace in the spatial sequence. For example, subspaces with significant long-range dependencies are given higher weights. This in turn adjusts the weights of each subspace in the current spatial sequence, ensuring that each subspace more accurately reflects the actual changes in the rock to be reconstructed.

[0075] By adjusting the weights, we strengthen the subspaces that have a significant impact on the global features while reducing the interference of other subspaces. Therefore, after adjusting the weights of the subspaces, we extract the global features of the image to obtain accurate global features. These global features contain the overall semantic information of the image, such as the shape and size of the rock to be reconstructed.

[0076] Step S103: extracting grain boundaries and microcracks in the image to obtain local features.

[0077] In this embodiment, a deep learning edge detection algorithm is used to perform convolution, weight adjustment and feature fusion operations on the image. After a series of operations, the feature expression ability of high-frequency details such as grain boundaries and microcracks in the image is enhanced. Then, the positions of high-frequency details such as grain boundaries and microcracks are determined, and local features containing information such as grain boundaries and microcracks are extracted.

[0078] The deep learning edge detection algorithm can be any of the following: a multi-layer rich feature hierarchy (RCF) detection algorithm, a holistically-nested edge detection (HED) algorithm, or a lightweight MobileNetV3 model algorithm. In this embodiment, the lightweight MobileNetV3 model algorithm is used as an example. The MobileNetV3 model algorithm is based on depthwise separable convolution and an inverted residual structure. The model first uses 3×3 convolutions in shallow layers to capture low-order edge features, such as grain boundaries and microcracks. It then uses multi-level inverted residual blocks to hierarchically refine edge features. A self-attention mechanism is introduced to enhance the response of important channels. An activation function is used to preserve high-frequency details, ultimately achieving precise detection of high-frequency details and obtaining accurate local features.

[0079] Step S104: fusing global features and local features to obtain a multi-scale joint embedding vector.

[0080] By combining features at different levels and scales, such as global features and local features, a more expressive joint embedding vector is generated. This vector is also called a multi-scale joint embedding vector.

[0081] In this embodiment, the specific fusion process mainly includes: first, adjusting the size of the local features to match the global features, then unifying the number of channels of the global features and local features, then flattening the global features and local features, splicing the flattened global features and local features to form a long vector, and then adjusting the weights of each vector through the attention mechanism to obtain a multi-scale joint embedding vector. The calculation formula is:

[0082] ,

[0083] Among them, G represents global features, L represents local features, Indicates that both G and L are dimensional vector, A is the data obtained by fusing global features and local features. Since the data contains features of multiple scales such as global and local, and both global features and local features are represented by vectors, the data is also called a multi-scale joint embedding vector.

[0084] It can be seen that the multi-scale joint embedding vector can not only describe the overall semantic information of the image, but also capture the high-frequency details in the image.

[0085] Step S105 : reconstructing the outline of the grain boundary according to the multi-scale joint embedding vector to obtain a reconstructed image.

[0086] First, the grain boundary contour offset is obtained. This contour offset can be obtained by pre-training the deformable convolution function. It is the error of the deformable convolution function due to the influence of its own parameters. The deformable convolution function is the key function for reconstructing the grain boundary contour. Therefore, it is necessary to adjust the grain boundary contour based on this contour offset. The specific calculation formula is:

[0087] ,

[0088] in, is the multi-scale joint embedding vector, is the image of the rock to be reconstructed, is the preset weight corresponding to the multi-scale joint embedding vector, F( ) is the deformable convolution function, and the deformable convolution function is used to adjust the multi-scale joint embedding vector according to the contour offset to obtain the input image , is the contour offset.

[0089] Then, take the input image As input to the generator, the generator receives the input image After that, at least one candidate image is output, and then the discriminator calls the standard image. The standard image refers to the image obtained by scanning the rock to be reconstructed through electronic equipment such as CT and SEM, and after reconstruction or other adjustment means, it clearly shows the mineral composition, crystal arrangement, pore distribution and other information inside the rock to be reconstructed. Based on the obtained candidate image and standard image, the discriminator determines the similarity between the candidate image and the standard image, and finally uses the candidate image with the largest similarity value as the reconstructed image. In this way, the obtained reconstructed image is constrained so that the microstructure of the rock to be reconstructed in the reconstructed image is more consistent with the actual situation. In addition, when there is only one candidate image, the discriminator uses the candidate image as the reconstructed image and outputs the similarity difference between the reconstructed image and the standard image, so that when the similarity difference exceeds the preset difference, the user is reminded to retrain and optimize the reconstruction model of the execution subject.

[0090] It should be noted that the execution subject is any one of the processor, server, terminal, and component, and the reconstruction model is an algorithm model used to implement the reconstruction method, which is composed of multiple network models.

[0091] Based on the microstructure reconstruction method based on deep learning of rock images provided in the above method embodiment, based on the same inventive concept, the embodiment of the present application also provides a microstructure reconstruction system based on deep learning of rock images, which is used to execute the above reconstruction method.

[0092] like Figure 2 As shown, the present application provides a microstructure reconstruction system based on deep learning of rock images, which consists of a data acquisition layer, a data extraction layer, a data processing layer and a data generation layer from top to bottom.

[0093] Data acquisition layer:

[0094] It is mainly used to exchange information with external terminal devices to obtain images of the rock to be reconstructed. Since the image of the rock to be reconstructed in this embodiment is an image obtained by scanning the rock to be reconstructed using an electronic device such as CT or SEM, the terminal device can be an electronic device with scanning capabilities such as CT or SEM, or an electronic device for storing images scanned by CT or SEM.

[0095] Data abstraction layer:

[0096] The data abstraction layer includes a first abstraction sublayer and a second abstraction sublayer arranged in parallel.

[0097] Among them, the first extraction sub-layer is mainly composed of the ViT (Vision Transformer) model, which is a deep learning model for image processing. Figure 3 and Figure 4 As shown, the ViT model is mainly used to implement the following operations:

[0098] Linear embedding labeling: Divide the image into N non-overlapping blocks P i , i=1, 2, ..., N. Assume that the size of each block is , then block P i Can be vectorized to x i , specifically: by linear projection x i Map to dimensional space, and obtain the subspace ,in is the projection weight matrix, The dimension is , which is used to transform the two-dimensional block P i The vector is converted to the specified dimensional space, is the bias vector, also expressed as dimensional vector, which is used to adjust the offset of the vector so that the ViT model can better learn the feature representation of the block.

[0099] Positional Encoding: via Learnable Positional Embeddings Capturing the spatial order of each subspace, the calculation formula is: , before capturing the spatial order of the subspace, a set of learnable classification labels needs to be added , according to the classification tag set Determine the basic spatial order z0, and then locate the spatial order of each subspace on the basic spatial order z0, so as to obtain the spatial order corresponding to each subspace By analogy, we can obtain the spatial order corresponding to all subspaces, and then output multiple subspaces as a spatial sequence according to the spatial order.

[0100] Weight adjustment: The self-attention mechanism is used to analyze the long-range dependencies between multiple subspaces in the spatial sequence, and the weights of each subspace are adjusted according to the long-range dependencies. In a specific example, the query vector, key vector, and value vector of the self-attention mechanism are represented by Q, K, and V respectively, and all three are set to dimensional vector, the calculation formula of the self-attention mechanism is:

[0101] , where K T Represents the transpose of the key vector.

[0102] If the self-attention mechanism is executed h times in parallel, we get:

[0103] ,

[0104] in, is the weight matrix, and the dimension supported by the self-attention mechanism is , the self-attention mechanism is used to project the query vector Q, key vector K and value vector V into different subspaces to learn feature relationships from different angles, such as the above The dimension is , it is possible to fuse outputs from multiple dimensions, so the self-attention mechanism is also called a multi-head self-attention mechanism, which can fuse multiple outputs. In this embodiment, the self-attention mechanism is executed in parallel 8 times, that is, h = 8. The self-attention mechanism can simultaneously capture long-range dependencies between subspaces from 8 different subspaces, thereby more comprehensively obtaining the spatial distribution and correlation information of crystals in the image.

[0105] In this embodiment, normalization layers are provided at both the front-end and back-end of the self-attention mechanism. The front-end normalization layer converts the idle sequence into a data format supported by the self-attention mechanism, while the back-end normalization layer converts the data output by the self-attention mechanism into a data format supported by the MLP Head.

[0106] Extracting global features: The MLP head is used to extract the reconstructed features of the image. The MLP head consists of a series of fully connected layers (FCLs). Each layer is connected by an activation function to form a multilayer perceptron (MLP). The MLP enhances the spatial distribution and correlation information of the crystals in the image, and then extracts the global features of the image based on this. Specifically, the structure of the MLP head is as follows:

[0107] ,

[0108] in, 、 are the preset weights of the fully connected layer, and , , Refers to datasets from the outside world, such as datasets from ImageNet, Wikipedia, etc. b1 and b2 are both preset biases. is the activation function.

[0109] It can be seen that the work of the first extraction sublayer implemented by the ViT model is as follows: First, the image is acquired. After the image is processed by linear embedding labels, it is divided into multiple blocks. Each block is mapped to a space of a specified dimension through linear projection to obtain the corresponding subspace. Secondly, position encoding adds learnable position information to each subspace to capture the spatial order relationship of the subspace and generate a spatial sequence containing position information. Thirdly, a self-attention mechanism is used to process the spatial sequence, including: first normalizing the input spatial sequence, then calculating the association weights of multiple subspaces in the spatial sequence through the self-attention mechanism, mining the long-range dependencies of the subspaces, and then adjusting the weights of the subspaces based on the long-range dependencies. Finally, the image after weight adjustment is normalized again and sent to the MLP Head for global feature extraction. The output of the MLP Head is fused with the front-end input through a residual connection to systematically extract the global features of the image.

[0110] like Figure 2 and Figure 3 As shown, the second extraction sublayer is primarily composed of a lightweight MobileNetV3 model. The MobileNetV3 model achieves a balance between lightweight and high performance by integrating a depthwise separable convolutional network, a nonlinear activation network, a squeeze-and-excitation (SE) network, and a residual network connection. In this embodiment, the MobileNetV3 model, based on the aforementioned components, performs the following operations:

[0111] First, the image undergoes a 1×1 convolution. By adjusting the number of convolution kernels, the number of channels is scaled to reduce the amount of computation. Then, an activation function is applied to optimize computational efficiency while retaining nonlinear expression. The specific calculation formula is:

[0112] .

[0113] Next, a 3×3 convolution kernel is used to independently extract features for each channel in the image. The output image has the same number of channels as the input, and its output is connected to a batch normalization network and a Hardswish activation function. The convolved image is downsampled and then passes through two fully connected layers, denoted by FC1 and FC2. The first fully connected layer, FC1, uses a ReLU activation function to add nonlinearity, while the second fully connected layer, FC2, uses a hard-σ activation function to generate information such as attention weights. The features obtained from the original input image and the processed image are added and fused using skip connections. Finally, a 1×1 convolution is performed to adjust the number of channels, and the final feature representation is output. In other words, after the image is processed by the multiple networks in the MobileNetV3 model, local features of high-frequency details such as grain boundaries and microcracks can be extracted.

[0114] It should be noted that for the image input by the data acquisition layer, the first extraction sublayer and the second extraction sublayer process the input image in parallel, and after processing, the first extraction sublayer outputs global features, while the second extraction sublayer outputs local features. The obtained global features and local features are uniformly input into the data processing layer for processing.

[0115] Data processing layer:

[0116] like Figure 2 As shown in Figure 2, the data processing layer is connected to the first extraction sublayer and the second extraction sublayer respectively. The data processing layer adopts a cross-modal attention mechanism to dynamically fuse global features and local features. The specific calculation formula is:

[0117] ,

[0118] Among them, G represents the global feature, L represents the local feature, and both G and L are dimensional vector, A is the data obtained by fusing global features and local features. Since the data contains features of multiple scales such as global and local, and both global features and local features are represented by vectors, the data is also called a multi-scale joint embedding vector.

[0119] It can be seen that the cross-modal attention mechanism calculates the similarity scores between global features and local features, and performs weighted adjustments on local features, so that the fused features can not only contain the spatial information of the crystal, but also highlight high-frequency details such as grain boundaries and microcracks, providing rich feature information for the subsequent data generation layer.

[0120] Data generation layer:

[0121] like Figure 3 As shown in Figure 1, the data generation layer is based on the improved U-Net++ model. Specifically, the improved U-Net++ model is to establish dense skip connections between the encoder and the decoder to enhance the multi-scale feature transfer, that is, to enhance the transfer of multi-scale joint embedding vectors. Figure 3 In the example, suppose the output feature of the i-th layer of the encoder is E i , the left dotted box represents the encoder, and the decoder layer j input feature is D j , the part symmetrical to the encoder represents the decoder. By setting dense skip connections between the encoder and decoder, D j It not only receives the output of the previous decoder layer, but also receives the features of the corresponding encoder layer and the intermediate layer, that is, , is the fusion function. Specifically, It can be a splicing operation, which splices multiple feature maps in the channel dimension, and then adjusts the number of channels and fuses features through 1×1 convolution. Figure 3 In the example, when the third layer output feature E3 of the encoder is fused with D2 after two layers of decoding, E3 and D2 are first concatenated according to the channel dimension to obtain a dimension of The feature map is then converted into a suitable dimension through 1×1 convolution to achieve effective transmission and fusion of multi-scale joint embedding vectors, thereby enhancing the U-Net++ model’s ability to utilize structural information at different scales, and enabling better restoration of the details and overall morphology of the mesostructure of the rock to be reconstructed during the reconstruction process.

[0122] Based on the improved U-Net++ model, the model also includes the following core components:

[0123] Deformable convolution: Deformable convolution is used to achieve dynamic upsampling and improve the accuracy of grain boundary contour reconstruction. The calculation formula of deformable convolution has been explained in step S105 of the method embodiment, so it will not be repeated here. It should be noted that in the process of grain boundary contour reconstruction, the fixed sampling position of the traditional convolution kernel is difficult to accurately capture the irregular shape of the grain boundary, while the deformable convolution obtains the contour offset through training and learning. , the convolution kernel can dynamically adjust the upsampling position during sampling, so that the outline of the grain boundary in the obtained input image is more consistent with the outline of the actual grain boundary.

[0124] Conditional Generative Adversarial Network: It mainly includes a generator and a discriminator. The generator receives the input image to obtain a candidate image. The discriminator determines the similarity between the candidate image and the standard image, and finally uses the candidate image with the largest similarity value as the reconstructed image.

[0125] It should be noted that when the generator generates candidate images, in addition to the input image, it also retrieves the noise vector and the microstructural characteristic parameters of the rock to be reconstructed. The noise vector is obtained through pre-training, and the characteristic parameters include but are not limited to data such as mineral content and crystal shape. Therefore, the generator generates candidate images based on multiple factors to ensure the accuracy of the resulting candidate images. When the discriminator determines the similarity between the candidate image and the standard image, it also considers the microstructural characteristic parameters of the rock to be reconstructed. If the characteristic parameters reflected by the candidate image and the standard image do not match, the lower the similarity value between the candidate image and the standard image, the more constrained the reconstructed image is, making the microstructure of the rock to be reconstructed in the reconstructed image more consistent with the actual situation.

[0126] In a possible implementation, the reconstruction system further includes a model training layer, which is used to train the reconstruction model obtained by combining the above-mentioned data acquisition layer, data extraction layer, data processing layer and data generation layer.

[0127] Before using the model training layer to train the reconstruction model, it is necessary to first build a multimodal training dataset, such as Figure 5 As shown, the process of constructing the data set is shown in steps S201 to S206.

[0128] Step S201, obtaining a SEM image.

[0129] First, a CT scan or SEM is used to scan the rock sample slices to obtain a clear SEM image. During the scanning process, the acceleration voltage, magnification, and other parameters of the electronic device are controlled to ensure that the image clearly shows the rock's microstructure, such as mineral crystals, grain boundaries, and microcracks.

[0130] Then, the SEM image is denoised and features enhanced respectively using the pre-processing method in step S101 in the method embodiment, and then the SEM image is saved.

[0131] Step S202: Count the mineral content in the SEM image. Identify the different mineral phases in the SEM image, segment the regions belonging to different mineral phases, and calculate the ratio of the number of pixels occupied by each mineral phase in the SEM image to obtain the corresponding mineral content. The specific calculation formula is:

[0132] ,

[0133] in, is the total number of pixels in the SEM image, is the number of pixels of one mineral phase, is the mineral content of this mineral phase.

[0134] Step S203: Identify the crystal shape and obtain characteristic parameters. Based on the digital image processing recognition algorithm, analyze the polygonal structure of the crystal in the SEM image, and extract the long axis of each polygonal structure. and short axis The length of , we get;

[0135] Isometric ratio ,λ is used to measure how close the crystal shape is to equiaxed;

[0136] Roundness , where s is the area of ​​the crystal polygonal structure, is the circumference of the crystal polygonal structure, roundness The closer it is to 1, the closer the shape is to a circle;

[0137] Sphericity , is the volume of the crystal, is the equivalent diameter of the crystal, sphericity Used to describe how close a crystal is to being a sphere;

[0138] In addition, the roughness rate of the crystal and characteristic parameters such as vertices and extreme points are also calculated. Among them, the roughness rate is obtained by analyzing the degree of change of the pixel points at the crystal boundary.

[0139] Based on the various characteristic parameters of the crystal obtained above, a probability distribution function of each characteristic parameter is established by statistically analyzing the characteristic parameters of a large number of crystal polygonal structures.

[0140] Step S204, counting grain boundary distribution: identifying and marking grain boundaries in the SEM image, counting information such as the length and area of ​​grain boundaries of different grain boundary types (such as coherent grain boundaries, semi-coherent grain boundaries, and incoherent grain boundaries), and calculating the proportion of grain boundaries of this type in the SEM image based on the area of ​​the grain boundaries. The proportion of this type of grain boundary in the overall grain boundary distribution is characterized by the size of the proportion.

[0141] Step S205 is crystal matching topology labels. Gradient boosting machines (GBMs) based on digital image recognition and random joint modeling are used to match topology labels. Topology labels include information such as crystal position, shape, contact relationships, and grain boundary types. The GBM model is an ensemble learning algorithm that improves model performance by combining multiple weak prediction models. This application uses the GBM model to implement directional modeling.

[0142] During directional modeling, grain boundaries are determined based on characteristic parameters of the crystal, such as mineral content, crystal shape characteristics, and grain boundary distribution constraints. Specifically, based on the pixel percentage of the mineral phase in the SEM image, each percentage is mapped to the modeling region, and the area of ​​each phase region is pre-allocated within the modeling region. Secondly, the phases in the modeling region are directionally adjusted based on the probability distribution function of each characteristic parameter. Then, based on the percentage of different grain boundary types, the spatial distribution probability of each grain boundary type is pre-defined. Each grain boundary is assigned a corresponding contact type label, and the grain boundary length and area distribution are adjusted to match the statistical results. Finally, a background mesh is generated based on Delaunay triangulation. Mineral content, crystal shape, and grain boundary type constraints are superimposed on the mesh. By quantitatively analyzing the statistical parameters of mineral content, crystal shape, and grain boundary distribution, a multidimensional mapping relationship is established. This mapping process uses Kriging interpolation technology to expand the two-dimensional characteristic parameters into three-dimensional space, ensuring the spatial continuity of the characteristic parameter distribution, thereby completing the construction of the boundaries of the mesostructure, namely the grain boundaries. Based on the obtained grain boundaries, polygons are constrained according to the shape of the crystal and the volume ratio of the crystal block is adjusted according to the mineral content to obtain a refined digital model. Each digital model is set with a corresponding topological label in advance. Therefore, after obtaining the digital model of the crystal, the topological label corresponding to the digital model is matched to match the obtained topological label as the topological label of the crystal.

[0143] The random joint modeling method is used for random modeling. It refers to sampling the characteristic parameters of the crystal to determine the shape of the polygon. For example, the volume proportion of the crystal is randomly determined according to the mineral content distribution, and the geometric structure of the crystal is constructed. At the same time, the spatial relationship of the crystal is considered. The polygon boundary is adjusted in combination with dynamic collision detection to avoid crystal overlap. The Monte Carlo method is then used to randomly assign the corresponding label type to each grain boundary, and the corresponding label proportion of each type of grain boundary is counted. The label with the highest proportion is used as the topological label of the grain boundary.

[0144] Based on the topological labels obtained by directional modeling and random modeling, a spatial blocking strategy is adopted to divide the modeling area into directional area and random area. The overlapping areas of the topological labels in directional modeling and random modeling are eliminated through the edge matching algorithm, and finally merged into a unified topological label set. The topological labels in the topological label set are stored in a hierarchical structure, including a directional label layer, a random label layer and an integrated label layer.

[0145] Step S206: Construct a multimodal training dataset. By integrating SEM images, feature parameters and topological labels, a multimodal training dataset is obtained, such as ,in, is the characteristic parameter, For SEM images, It is the topology label.

[0146] Based on the obtained multimodal training dataset , randomly split the multimodal training dataset There are three groups, one of which is the training group, and the other two are the control group and the verification group. The training group and the control group are input into the model training layer, and the model training layer trains the reconstructed model based on the data of the training group and the control group.

[0147] like Figure 2 As shown, the model training layer mainly includes the first training sublayer, the second training sublayer and the third training sublayer connected in series.

[0148] Among them, the first training sublayer is used to construct a loss function, specifically: a multi-task loss function such as the pixel-level reconstruction loss of the joint image, the grain boundary contrast loss (Dice coefficient), and the regression loss of the feature parameters to optimize the training process of the reconstruction model.

[0149] The pixel-level reconstruction loss combines the structural similarity index (SSIM index) and the absolute error loss (L1). The SSIM index is an indicator used to measure the similarity between two digital images. When one of the two images is an undistorted image, also known as a standard image, and the other is a distorted image, also known as a reconstructed image, the structural similarity between the two can be regarded as an image quality measurement indicator of the distorted image. In this embodiment, the loss function calculation formula of the SSIM index is:

[0150] ,

[0151] in, is a standard image, To reconstruct the image, , and Standard images and reconstructed image The mean of is used to measure the average brightness of the image. and are the standard deviations of the standard image and the reconstructed image, respectively. The standard deviation is used to reflect the discrete degree of the image pixel values. That is, the larger the standard deviation, the richer the image details; is the covariance between the standard image and the reconstructed image, which is used to reflect the correlation between the pixel value changes of the two; and is a constant used to maintain stability, L is the dynamic range of pixel values ​​(for 8-bit images, L = 255), and are all preset values. In this embodiment, , During training, it is necessary to continuously adjust the SSIM index loss function to ensure the calculated loss value The closer it is to 0, the closer the reconstructed image is to the standard image.

[0152] L1 is used to calculate the absolute difference between the reconstructed image and the standard image. The calculation formula of the L1 loss function is:

[0153] ,

[0154] in, and are the height and width of the image, and The standard image and the reconstructed image are The pixel value at the location.

[0155] It can be seen that the L1 loss function can make the pixel values ​​of the reconstructed image as close as possible to the pixel values ​​of the standard image, which helps to improve the accuracy of the reconstructed image.

[0156] The Dice coefficient loss function is mainly used to measure the degree of overlap between the grain boundaries in the reconstructed image and the grain boundaries in the standard image. The calculation formula of the Dice coefficient loss function is:

[0157] ,

[0158] Wherein, A is the pixel set of the grain boundary in the standard image, and B is the pixel set of the grain boundary in the reconstructed image. represents the number of pixels where the grain boundaries in the standard image overlap with those in the reconstructed image. |A| and |B| are the total number of pixels at the grain boundaries in the standard image and the reconstructed image, respectively. In this embodiment, by controlling the value of the Dice coefficient loss function to approach 0, the accuracy of grain boundary contour reconstruction can be effectively improved by ensuring that the grain boundaries in the reconstructed image match those in the standard image.

[0159] The regression loss of feature parameters mainly uses the mean absolute error (MAE) loss function, which is used to evaluate the accuracy of the reconstruction model in identifying feature parameters. Specifically, the calculation formula of the MAE loss function is:

[0160] ,

[0161] in, is the characteristic parameter in the standard image, is the characteristic parameter in the reconstructed image, This embodiment controls the value of the MAE loss function to be continuously close to 0, thereby adjusting the average deviation between the reconstructed image output by the reconstruction model and the standard image to be smaller, which helps to ensure that the reconstruction model can accurately reflect the characteristic parameters of the rock mesostructure.

[0162] Based on the above multiple loss functions, in order to control the optimization direction of the reconstruction model from the overall perspective, this embodiment also provides a total loss function:

[0163] ,

[0164] in 、 、 、 They are all weight coefficients, which can be adaptively adjusted according to the importance of each loss function, and this embodiment does not impose any restrictions.

[0165] The second training sublayer primarily uses a five-fold cross-validation method to evaluate the accuracy of the reconstructed images obtained by the reconstruction model. In this embodiment, the reconstruction model is trained with a training group, and then the performance of the reconstruction model is evaluated with a control group. Various evaluation indicators are recorded. This cycle is repeated at least five times, and the average of the five evaluation results is selected as the final performance indicator of the reconstruction model to more accurately evaluate the generalization ability and reconstruction accuracy of the reconstruction model.

[0166] The third training sublayer primarily optimizes the parameters of the reconstruction model. During the aforementioned training process, the network structure and training parameters of the reconstruction model are adjusted. These include, but are not limited to, the size of the subspace in the ViT model, the specified projection dimension, and the number of dimensions of attention in the multi-head self-attention mechanism; the number of layers and channels in the MobileNetV3 network; and the number of encoder and decoder layers, convolution kernel size, and skip connection structure in the U-Net++ architecture. Training parameters also include, but are not limited to, the loss value, the number of iterations, and the iteration duration. After repeated adjustments to the reconstruction model parameters, the optimal set of parameters is selected as the reconstruction model parameters, enabling the reconstruction model to learn the microstructure of the rock to be reconstructed and reconstruct the microstructure in the image, while also ensuring the accuracy of the reconstructed image.

[0167] It should be noted that the data layers, models, etc. involved in the above embodiments are all software programs, which are used in the execution body for processing the image of the rock to be reconstructed.

[0168] It should also be noted that the order of execution of the steps in the above embodiments does not necessarily imply a specific order of execution. The order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In practical applications, all the above possible implementation methods can be combined in any way to form possible embodiments of the present application, and will not be described in detail here.

[0169] Based on the same inventive concept, an embodiment of the present application also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute any one of the above-mentioned embodiments of the microstructure reconstruction method based on deep learning of rock images.

[0170] In an exemplary embodiment, an electronic device is provided, such as Figure 6 As shown, Figure 6 The electronic device 600 shown includes a processor 601 and a memory 603. The processor 601 and the memory 603 are connected, for example, via a bus 602. Optionally, the electronic device 600 may further include a transceiver 604. It should be noted that in actual applications, the number of transceivers 604 is not limited to one, and the structure of the electronic device 600 does not constitute a limitation on the embodiments of the present application.

[0171] Processor 601 may be a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 601 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0172] Bus 602 may include a path for transmitting information between the above components. Bus 602 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 602 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0173] The memory 603 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0174] The memory 603 is used to store computer program codes for executing the solution of the present application, and the execution is controlled by the processor 601. The processor 601 is used to execute the computer program codes stored in the memory 603 to implement the contents shown in the above method embodiment.

[0175] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0176] Based on the same inventive concept, an embodiment of the present application also provides a storage medium, which stores a computer program, wherein the computer program is configured to execute the microstructure reconstruction method based on deep learning of rock images of any of the above embodiments when running.

[0177] Those skilled in the art will clearly understand that the specific working processes of the systems, devices, and modules described above can refer to the corresponding processes in the aforementioned method embodiments, and for the sake of brevity, they will not be further described here.

[0178] Those skilled in the art will understand that the technical solution of the present application, in essence, or in whole or in part, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of program instructions that cause an electronic device (such as a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application when the program instructions are executed. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0179] Alternatively, all or part of the steps of implementing the aforementioned method embodiments may be accomplished by hardware related to program instructions (such as electronic devices such as personal computers, servers, or network devices), and the program instructions may be stored in a computer-readable storage medium. When the program instructions are executed by a processor of an electronic device, the electronic device executes all or part of the steps of the methods described in the various embodiments of the present application.

[0180] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that, within the spirit and principles of the present application, they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate from the protection scope of the present application.

Claims

1. A microstructure reconstruction method based on deep learning of rock images, characterized in that: include: Acquire an image of the rock to be reconstructed; After dividing the image into blocks and mapping them into a space of a specified dimension, a plurality of subspaces are obtained, and a global feature is obtained based on a long-range dependency relationship between the plurality of subspaces; Extracting grain boundaries and microcracks in the image to obtain local features, wherein the grain boundaries are boundaries of crystals in the image, the crystals are components of the rock to be reconstructed, and the microcracks are gaps between two adjacent crystals; fusing the global features and the local features to obtain a multi-scale joint embedding vector; Reconstructing the contour of the grain boundary according to the multi-scale joint embedding vector to obtain a reconstructed image, including: Obtaining a contour offset of the grain boundary; Adjusting the multi-scale joint embedding vector according to the contour offset to obtain an input image includes: ,in, is the input image, is the multi-scale joint embedding vector, is the image of the rock to be reconstructed, is the preset weight corresponding to the multi-scale joint embedding vector, F( ) is a deformable convolution function, and the deformable convolution function is used to adjust the multi-scale joint embedding vector according to the contour offset to obtain the input image, is the contour offset; The reconstructed image is obtained according to the input image and a standard image, wherein the standard image refers to an image showing the mineral composition, crystal arrangement, and pore distribution of the rock to be reconstructed.

2. The method according to claim 1, characterized in that The step of obtaining the reconstructed image according to the input image and the standard image includes: Using a generator to obtain at least one candidate image according to the input image; Using a discriminator to obtain the similarity between the candidate image and the standard image based on the candidate image; The candidate image with the highest similarity is selected as the reconstructed image.

3. The method according to claim 1, characterized in that After obtaining the image of the rock to be reconstructed, the method further includes: Using a Gaussian filtering algorithm to filter out interference signals in the image; A histogram equalization method is used to enhance the contrast of the image.

4. A microstructure reconstruction system based on deep learning of rock images, used to execute the method according to any one of claims 1 to 3, characterized in that: include: A data acquisition layer, used to obtain images of rocks to be reconstructed; a data extraction layer, comprising a first extraction sublayer and a second extraction sublayer arranged in parallel, wherein the first extraction sublayer is used to extract global features of the image, and the second extraction sublayer is used to extract local features of the image; a data processing layer, connected to the first extraction sublayer and the second extraction sublayer, respectively, for fusing the global features and the local features to obtain the multi-scale joint embedding vector; A data generation layer is used to obtain a reconstructed image according to the multi-scale joint embedding vector.

5. The system according to claim 4, characterized in that The first extraction sublayer uses a ViT model to extract global features of the image, and the ViT model is used to perform the following operations: Linear embedding labeling: the image is divided into blocks and mapped to a space of specified dimensions to obtain multiple subspaces; Position encoding: After capturing the spatial order of multiple subspaces, a spatial sequence is generated; Weight adjustment: adjusting the weight of each subspace according to the long-range dependencies between the multiple subspaces in the spatial sequence; Extract global features.

6. The system according to claim 4, characterized in that The second extraction sublayer uses the MobileNetV3 model to extract local features of the image; The MobileNetV3 model includes at least a separable convolutional network, a nonlinear activation network and a residual network.

7. The system according to any one of claims 5 or 6, characterized in that The data processing layer adopts a cross-modal attention mechanism to fuse the global features and the local features; The calculation formula of the cross-modal attention mechanism is: , Among them, G represents the global feature, L represents the local feature, and both G and L are dimensional vector, A is the multi-scale joint embedding vector, is the function of the cross-modal attention mechanism.

8. The system according to claim 4, wherein: The system further includes a model training layer, wherein the model training layer is used to train a reconstruction model, wherein the reconstruction model is obtained by combining the data acquisition layer, the data extraction layer, the data processing layer, and the data generation layer; The model training layer includes: A first training sublayer is used to construct a loss function, wherein the loss function is used to constrain the reconstruction model; The second training sublayer uses a five-fold cross-validation method to evaluate the accuracy of the reconstructed image output by the reconstruction model; The third training sublayer: adjusts the parameters of the reconstruction model and selects the best set of parameters as the parameters of the reconstruction model.

Images