A sandstone microscopic image segmentation method and device

Abstract

The embodiments of this specification relate to the field of image processing technology, and in particular to a method and device for segmenting sandstone microscopic images. The method includes: obtaining a sandstone microscopic image to be segmented, and a segmentation text prompt and a segmentation space prompt corresponding to the sandstone microscopic image; inputting the sandstone microscopic image into a pre-trained image segmentation model for calculation to obtain a segmentation result of the sandstone microscopic image. The embodiments of this specification achieve accurate and efficient segmentation of mineral particles, rock fragments, cementing materials and pore structures in sandstone microscopic images, which is helpful for analyzing sandstone composition and microstructural characteristics. The method of the embodiments of this specification has high application value in the exploration and evaluation of oil and gas resource reservoirs.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of image processing technology, and in particular to a sandstone microscopic image segmentation method and device. Background Art

[0002] Sandstone thin section identification is the basis of reservoir formation mechanism, rock physics modeling and sweet spot prediction. Most traditional identification methods require manual identification of mineral components and microstructures in thin sections through polarizing microscopes and naked eyes, which requires the appraiser to have rich knowledge of petrology and mineralogy, is inefficient and highly subjective. In recent years, with the development of deep learning technology, intelligent methods have been increasingly widely studied in the field of rock thin section identification, among which convolutional neural networks (CNN), residual connection networks (ResNet) and U-Net networks are the most popular. Compared with traditional methods, such intelligent identification methods have greatly improved the efficiency and accuracy of rock thin section identification. However, deep neural network models are usually set for a single task, and the sample types annotated during the training process are limited. When the model encounters data that has not been trained, it is often difficult to output accurate results. In addition, training such models usually requires a large number of rock thin section samples to give full play to their advantages, so sample collection and annotation are also a problem.

[0003] How to efficiently segment sandstone microscopic images is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In order to solve the problem that sandstone microscopic images cannot be efficiently segmented in the prior art, the embodiments of this specification provide a sandstone microscopic image segmentation method and device, which realizes accurate and efficient segmentation of mineral particles, rock fragments, cementing materials and pore structures in sandstone microscopic images, and is helpful for analyzing the composition and microstructural characteristics of sandstone. The method of the embodiments of this specification has high application value in the exploration and evaluation of oil and gas resource reservoirs.

[0005] The specific technical solutions of the embodiments of this specification are as follows:

[0006] On the one hand, an embodiment of this specification provides a sandstone microscopic image segmentation method, comprising:

[0007] Acquire a sandstone microscopic image to be segmented, and a segmentation text prompt and a segmentation space prompt corresponding to the sandstone microscopic image;

[0008] The sandstone microscopic image is input into a pre-trained image segmentation model for calculation, wherein the image segmentation model includes an image encoder, a text prompt encoder, a spatial prompt encoder, a semantic spatial prompt fuser and a mask decoder, wherein the image encoder encodes the sandstone microscopic image to generate an image embedding vector; the text prompt encoder encodes the segmentation text prompt to generate an initial semantic prompt embedding, and performs weighted calculation on the initial semantic prompt embedding through semantic prompt learning to obtain a target semantic prompt embedding; the spatial prompt encoder encodes the segmentation spatial prompt to generate an initial spatial prompt embedding, and performs weighted calculation on the initial spatial prompt embedding through spatial prompt learning to obtain a target spatial prompt embedding; the semantic spatial prompt fuser fuses the target semantic prompt embedding and the target spatial prompt embedding to obtain a semantic spatial fused prompt embedding; the mask decoder decodes the image embedding vector, the segmentation text prompt, the segmentation spatial prompt and the semantic spatial fused prompt embedding to obtain a segmentation result of the sandstone microscopic image.

[0009] Furthermore, the step of training the image segmentation model includes:

[0010] Inputting a training image in a training image data set into an image encoder for encoding to obtain a training image embedding vector corresponding to the training image;

[0011] The segmented text prompt labels corresponding to the training images in the training image data set are input into the text prompt encoder for calculation, firstly a default semantic prompt embedding corresponding to the segmented text prompt label is generated, and then the default semantic prompt embedding is weightedly calculated through semantic prompt learning to obtain a training semantic prompt embedding;

[0012] Inputting the segmentation spatial cue labels corresponding to the training images in the training image data set into the spatial cue encoder for calculation, first generating a default spatial cue embedding corresponding to the segmentation spatial cue label, and then performing weighted calculation on the default spatial cue embedding through spatial cue learning to obtain a training spatial cue embedding;

[0013] Inputting the training semantic hint embedding and the training spatial hint embedding into the semantic spatial hint fuser for calculation to obtain the training semantic spatial fusion hint embedding;

[0014] The training image embedding vector, segmentation text prompt label, segmentation space prompt label and training semantic space fusion prompt embedding corresponding to each training image are input into the mask decoder for decoding to obtain the segmentation result corresponding to each training image, and the loss value is calculated according to the segmentation result and the actual result. When the loss value does not meet the requirements, the parameters of the image encoder, text prompt encoder, space prompt encoder, semantic space prompt fusion and mask decoder are adjusted, and the step of inputting the training images in the training image data set into the image encoder for encoding to obtain the training image embedding vector corresponding to the training image is repeatedly executed until the loss value meets the requirements and the trained image segmentation model is obtained.

[0015] Furthermore, each attention module of the image encoder includes two adapters, the first adapter is placed after the multi-head attention layer and before the residual connection, the second adapter is placed on the residual path of the multi-layer perceptron layer after the multi-head attention layer, and the scaling factor is used to perform scaling embedding after the second adapter;

[0016] The adapter consists of three parts: up-projection, ReLU activation function and down-projection;

[0017] The adapter's processing steps include:

[0018] The input data is down-projected to reduce its dimension, the reduced dimension result is activated by the ReLU activation function, and then the activated result is up-projected to restore its original dimension.

[0019] Furthermore, the default semantic hint embedding is weighted by semantic hint learning, and the formula for training semantic hint embedding is obtained as follows:

[0020]

[0021] in, represents training semantic cue embedding, represents the cth weight, represents C learnable weights, represents the cth semantic embedding vector, represents the C learnable semantic embedding vectors corresponding to the training images, Represents the default semantic hint embedding.

[0022] Furthermore, the default spatial cue embedding is weighted by spatial cue learning to obtain the formula for training spatial cue embedding:

[0023]

[0024] in, represents the training spatial cue embedding, represents the nth weight, represents N learnable weights, represents the nth spatial embedding vector, represents the N learnable spatial embedding vectors corresponding to the training images, Represents the default space hint embedding.

[0025] Furthermore, the training semantic hint embedding and the training spatial hint embedding are input into the semantic spatial hint fuser for calculation, and the training semantic spatial fusion hint embedding formula is obtained as follows:

[0026]

[0027] Among them, Z SSPL represents the training semantic space fusion hint embedding, Dense represents the fully connected layer, and D represents the target dimension of the fully connected layer conversion. Indicates that and Concatenates two vectors into one vector.

[0028] Furthermore, each attention module of the mask decoder includes a high-level hint adapter, and the high-level hint adapter is placed between two residual connections.

[0029] Further, the high-level cue adapter consists of four parts: down-projection, high-level cue embedding, ReLU activation function and up-projection;

[0030] Inputting the training image embedding vector, segmentation text prompt label, segmentation space prompt label and training semantic space fusion prompt embedding corresponding to each training image into the mask decoder for decoding, and obtaining the segmentation result corresponding to each training image further includes:

[0031] The segmented text prompt labels and segmented space prompt labels are integrated through the normalization layer, and then the integrated results are reduced in dimension through down-projection to obtain down-projection embedding, whose formula is: Among them, Z down represents the down-projection output, Down represents the down-projection operation, and Norm represents the normalization layer. Indicates the segmentation text prompt label, Indicates the split space prompt label;

[0032] The training semantic space fusion prompt embedding is calculated through the MLP multi-layer perceptron and reshaping operation to generate the weights applied to the high-level prompt adapter, which is formulated as: W SSPL =Re(MLP(ZSSPL )), where W SSPL represents weight, Re represents reshaping operation, and MLP represents multi-layer perceptron;

[0033] Weight W SSPL and the lower projection output Z down Perform matrix multiplication, and then normalize the product to obtain the result of high-level prompt embedding, then activate the result of high-level prompt embedding through the ReLU activation function, and then project the activation result up to restore its original dimension to obtain the up-projection output. The formula is: Among them, Z up represents the up-projection output, Up represents the up-projection operation, and Norm represents the normalization layer. Represents matrix multiplication;

[0034] Through the formula {M, S} = Decoder (Z I |Z up ) Upward projection output Z up And the corresponding training image embedding vector is decoded to obtain the segmentation result, where M is the predicted segmentation mask, S is the corresponding confidence score of the predicted segmentation mask, and Decoder represents the decoding operation.

[0035] Furthermore, adjusting the parameters of the image encoder, the text hint encoder, the space hint encoder, the semantic space hint fuser and the mask decoder further includes:

[0036] adjusting parameters of two adapters in the image encoder;

[0037] Adjust the weights W in the textual prompt encoder T ;

[0038] Adjust the weights W in the spatial cue encoder S ;

[0039] Adjusting the parameters of the fully connected layer Dense in the semantic space hint fuser;

[0040] Parameters of the advanced hint adapter in the mask decoder are adjusted.

[0041] On the other hand, the embodiments of this specification also provide a sandstone microscopic image segmentation device, the device comprising:

[0042] A data acquisition unit, used for acquiring a sandstone microscopic image to be segmented, and a segmentation text prompt and a segmentation space prompt corresponding to the sandstone microscopic image;

[0043] The segmentation unit is used to input the sandstone microscopic image into a pre-trained image segmentation model for calculation, wherein the image segmentation model includes an image encoder, a text prompt encoder, a spatial prompt encoder, a semantic spatial prompt fuser and a mask decoder, wherein the image encoder encodes the sandstone microscopic image to generate an image embedding vector; the text prompt encoder encodes the segmentation text prompt to generate an initial semantic prompt embedding, and performs weighted calculation on the initial semantic prompt embedding through semantic prompt learning to obtain a target semantic prompt embedding; the spatial prompt encoder encodes the segmentation spatial prompt to generate an initial spatial prompt embedding, and performs weighted calculation on the initial spatial prompt embedding through spatial prompt learning to obtain a target spatial prompt embedding; the semantic spatial prompt fuser fuses the target semantic prompt embedding and the target spatial prompt embedding to obtain a semantic spatial fused prompt embedding; the mask decoder decodes the image embedding vector, the segmentation text prompt, the segmentation spatial prompt and the semantic spatial fused prompt embedding to obtain a segmentation result of the sandstone microscopic image.

[0044] Using the embodiment of this specification, the sandstone microscopic image to be segmented and the segmentation text prompt and segmentation space prompt corresponding to the sandstone microscopic image are taken as input, the sandstone microscopic image is encoded using the image encoder in the trained image segmentation model, the segmentation text prompt is encoded using the text prompt encoder in the trained image segmentation model, the segmentation space prompt is encoded using the space prompt encoder in the trained image segmentation model, the semantic space fusion prompt embedding is calculated using the trained semantic space prompt fuser, and then the image embedding vector, the segmentation text prompt, the segmentation space prompt and the semantic space fusion prompt embedding are input into the decoder for decoding. The decoder includes calculating the semantic space fusion cue embedding based on the parameters of the trained high-level cue adapter to obtain the weight of the high-level cue adapter, and then using the weight to fuse the semantic cue embedding and the spatial cue embedding. Therefore, the trained high-level cue adapter realizes the mutual fusion of spatial information and semantic information, thereby utilizing the synergistic fusion of spatial information and semantic information to better prompt the image segmentation model, thereby realizing accurate and efficient segmentation of mineral particles, rock fragments, cement and pore structures in sandstone microscopic images, which is helpful for analyzing the composition and microstructural characteristics of sandstone, and solves the problem that sandstone microscopic images cannot be efficiently segmented in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a schematic diagram of an implementation system of a sandstone microscopic image segmentation method in an embodiment of this specification;

[0047] Figure 2a It is a schematic diagram of a flow chart of a sandstone microscopic image segmentation method according to an embodiment of this specification;

[0048] Figure 2b It is a detailed flow chart of inputting the sandstone microscopic image into a pre-trained image segmentation model for calculation to obtain the segmentation result of the sandstone microscopic image in the embodiment of this specification;

[0049] Figure 3 Shown is a schematic diagram of the process of training the image segmentation model in the embodiment of this specification;

[0050] Figure 4 The figure is a schematic diagram of the structure of the image encoder in the embodiment of this specification;

[0051] Figure 5 The figure shows the overall architecture of the semantic-spatial cue fusion process in the embodiments of this specification;

[0052] Figure 6 It is a schematic diagram of the structure of an advanced prompt adapter integrating semantic and spatial prompts in an embodiment of this specification;

[0053] Figure 7 It is a schematic diagram of the structure of a mask decoder guided by advanced hints in an embodiment of this specification;

[0054] Figure 8 The figure shows a schematic diagram of the structure of a sandstone microscopic image segmentation device in an embodiment of this specification;

[0055] Fig. 9 The figure is a schematic diagram of the structure of a computer device in an embodiment of the present specification.

[0056]

Description of reference numerals

[0057] 101. Terminal;

[0058] 102. Server;

[0059] 801, data acquisition unit;

[0060] 802, segmentation unit;

[0061] 902. Computer equipment;

[0062] 904. Processing equipment;

[0063] 906. Storage resources;

[0064] 908, drive system;

[0065] 910, input / output module;

[0066] 912. Input device;

[0067] 914. Output device;

[0068] 916. Presentation equipment;

[0069] 918. Graphical user interface;

[0070] 920, network interface;

[0071] 922, communication link;

[0072] 924. Communication bus. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of this specification.

[0074] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of this specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0075] It should be noted that the acquisition, storage, use, and processing of data in the technical solutions of the embodiments of this specification comply with the relevant provisions of national laws and regulations.

[0076] It should be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0077] like Figure 1 The diagram is a schematic diagram of a system for implementing a sandstone microscopic image segmentation method in an embodiment of the present specification, including a terminal 101 and a server 102. The terminal 101 and the server 102 can communicate with each other through a network, and the network can include a local area network (LAN), a wide area network (WAN), the Internet or a combination thereof, and is connected to a website, a user device (such as a computing device) and a back-end system.

[0078] The user can input the sandstone microscopic image to be segmented and the corresponding segmentation text prompt and segmentation space prompt to the server 102 through the terminal 101. The server 102 stores a trained image segmentation model. The sandstone microscopic image, the segmentation text prompt and the segmentation space prompt are calculated using the trained image segmentation model, so as to obtain the segmentation result of the sandstone microscopic image under the segmentation text prompt and the segmentation space prompt, and the segmentation result can be provided to the user through the terminal 101.

[0079] Alternatively, the server 102 may be a node of a cloud computing system (not shown), or each server may be a separate cloud computing system including a plurality of computers interconnected by a network and operating as a distributed processing system.

[0080] In addition, it should be noted that Figure 1 What is shown is only an application environment provided by the embodiment of this specification. In actual application, it can also be applied to other environments, and this specification does not limit it.

[0081] The model used in the method for segmenting sandstone microscopic images in the embodiments of this specification is a RockSAM model suitable for sandstone microscopic image segmentation obtained by improving the traditional SAM model. The RockSAM model trained in the embodiments of this specification is formed into an application in a computer system by writing code, etc. The computer system can obtain the sandstone microscopic image to be segmented and its corresponding segmentation text prompts and segmentation space prompts, and then input the sandstone microscopic image into the application for calculation, and output the segmentation result of the sandstone microscopic image.

[0082] The traditional SAM model consists of three components: ① Image encoder: The image encoder is responsible for extracting input image features and converting them into image embeddings. The image encoder uses 14×14 window attention and four equidistant global attention blocks. The output of the image encoder is a 16-fold downsampled image embedding of the input image; ② Prompt encoder: It includes a text prompt encoder and a spatial prompt encoder, which are used to integrate user interactive prompts, and convert input text prompts and spatial prompts into prompt embeddings respectively. The text prompt is a text description of the segmented object, and the spatial prompt contains manually provided points, borders and masks; ③ Mask decoder: Its function is to return the expected segmentation mask based on the image embedding and prompt embedding.

[0083] Since SAM has been trained on a large dataset of more than 11 million images and 1 billion masks, it is very powerful and can achieve universal image segmentation with zero samples. At the same time, SAM also introduces an interactive prompt segmentation mode, in which some visual cues, such as points, borders, and masks, are provided to the model to indicate the area that the model needs to segment in the image, and the model can easily segment the objects in the area. However, if SAM is directly applied to the segmentation of sandstone microscopic images, the effect is not ideal. First, because sandstone contains a large number of colorless and transparent mineral particles with similar refractive index, the color or grayscale presented under the microscope is very similar, and the mineral particles in sandstone are numerous, densely distributed, irregular in shape, inconsistent in size, and diverse in texture. The boundary between the pores is very fuzzy. For quartz minerals and feldspar minerals with large particle size, obvious edges, and colorless and transparent, SAM can usually achieve good segmentation, but it is relatively difficult to segment dark minerals. At the same time, there are a lot of cements, rock fragments and pores in sandstone, and their shapes are very irregular. The cements are filled between the rock fragments and bonded together with the rock fragments, which is very difficult to distinguish. The pore structure is blurred due to dissolution, which further limits the application of SAM. In addition, it is very important that the technical defects of the SAM model itself also weaken the segmentation effect on sandstone microscopic images. Because the default text prompt encoder of SAM can usually only learn the foreground knowledge of the image well, but the learning of background knowledge is insufficient; and the default spatial prompt encoder uses XY coordinates as spatial prompts, but there is a limitation when learning such spatial prompts in low-dimensional space (i.e., two-dimensional space in the XY coordinate system), that is, its search space is limited, usually limited by the number and distribution of grid points, which means that the spatial prompt encoder may not be able to effectively process very complex or high-dimensional spatial data. These two defects also limit the segmentation effect of SAM. In SAM, prompts are crucial for segmentation, and slight changes in prompts can easily affect the segmentation results.

[0084] In view of the difficulties in segmenting the above-mentioned sandstone microscopic images and the technical defects of the SAM model itself, the embodiments of this specification provide a sandstone microscopic image segmentation method, which realizes the accurate and efficient segmentation of mineral particles, rock fragments, cement and pore structure in sandstone microscopic images, and is helpful for analyzing the composition and microstructural characteristics of sandstone. The method of the embodiments of this specification has high application value in the exploration and evaluation of oil and gas resource reservoirs. Figure 2a and Figure 2b The figure shows a flow chart of a sandstone microscopic image segmentation method according to an embodiment of the present specification. The figure describes the process of segmenting a sandstone microscopic image. The sequence of steps listed in the embodiment is only one of many execution sequences of steps and does not represent the only execution sequence. When the actual system or device product is executed, it can be executed in the order or in parallel according to the method shown in the embodiment or the accompanying drawings. Figure 2a As shown, the method may be executed by the server 102, and may include:

[0085] Step 201: obtaining a sandstone microscopic image to be segmented, and a segmentation text prompt and a segmentation space prompt corresponding to the sandstone microscopic image;

[0086] Step 202: input the sandstone microscopic image into a pre-trained image segmentation model for calculation to obtain a segmentation result of the sandstone microscopic image.

[0087] The image segmentation model includes an image encoder, a text hint encoder, a space hint encoder, a semantic space hint fusion device and a mask decoder.

[0088] Figure 2b The figure shows a detailed flow chart of inputting the sandstone microscopic image into a pre-trained image segmentation model for calculation to obtain the segmentation result of the sandstone microscopic image in the embodiment of this specification, which may include the following steps:

[0089] Step 2021: the image encoder encodes the sandstone microscopic image to generate an image embedding vector;

[0090] Step 2022: the text prompt encoder encodes the segmented text prompt to generate an initial semantic prompt embedding, and performs weighted calculation on the initial semantic prompt embedding through semantic prompt learning to obtain a target semantic prompt embedding;

[0091] Step 2023: the spatial cue encoder encodes the segmented spatial cue to generate an initial spatial cue embedding, and performs weighted calculation on the initial spatial cue embedding through spatial cue learning to obtain a target spatial cue embedding;

[0092] Step 2024: the semantic space cue fuser fuses the target semantic cue embedding and the target space cue embedding to obtain a semantic space fused cue embedding;

[0093] Step 2025: The mask decoder decodes the image embedding vector, the segmentation text prompt, the segmentation space prompt and the semantic space fusion prompt embedding to obtain the segmentation result of the sandstone microscopic image.

[0094] Using the embodiment of this specification, the sandstone microscopic image to be segmented and the segmentation text prompt and segmentation space prompt corresponding to the sandstone microscopic image are taken as input, the sandstone microscopic image is encoded using the image encoder in the trained image segmentation model, the segmentation text prompt is encoded using the text prompt encoder in the trained image segmentation model, the segmentation space prompt is encoded using the space prompt encoder in the trained image segmentation model, the semantic space fusion prompt embedding is calculated using the trained semantic space prompt fuser, and then the image embedding vector, the segmentation text prompt, the segmentation space prompt and the semantic space fusion prompt embedding are input into the decoder for decoding. The decoder includes calculating the semantic space fusion cue embedding based on the parameters of the trained high-level cue adapter to obtain the weight of the high-level cue adapter, and then using the weight to fuse the semantic cue embedding and the spatial cue embedding. Therefore, the trained high-level cue adapter realizes the mutual fusion of spatial information and semantic information, thereby utilizing the synergistic fusion of spatial information and semantic information to better prompt the image segmentation model, thereby realizing accurate and efficient segmentation of mineral particles, rock fragments, cement and pore structures in sandstone microscopic images, which is helpful for analyzing the composition and microstructural characteristics of sandstone, and solves the problem that sandstone microscopic images cannot be efficiently segmented in the prior art.

[0095] In the embodiments of this specification, after the image segmentation results are obtained, the components and structural characteristics of the sandstone corresponding to the sandstone microscopic image are analyzed according to the segmentation results, thereby guiding the exploration and evaluation of the oil and gas resource reservoir in the formation where the sandstone is located according to the components and structural characteristics of the sandstone.

[0096] The image segmentation model in step 202 is the RockSAM model improved in the embodiment of this specification, which can achieve better results in the segmentation of sandstone microscopic images.

[0097] The RockSAM model consists of the following components:

[0098] Image Encoder: An image encoder with an integrated adapter module that can more efficiently convert input images into image embedding vectors, making it easier to adapt to sandstone microscopic image segmentation tasks.

[0099] Text Hint Encoder: An encoder that converts user text descriptions of mineral grains, rock fragments, cements, and pore structures in microscopic images into semantic hint embeddings.

[0100] Spatial Cue Encoder: An encoder that converts user-provided rectangular bounding boxes into spatial cue embeddings.

[0101] Semantic-Spatial Cue Fusor: A fuser that simultaneously learns semantic and spatial cues to provide more appropriate cue embeddings for segmentation tasks.

[0102] Mask Decoder: A mask decoder integrated with high-level cues that can more accurately predict segmentation masks in combination with image embeddings under the guidance of high-level cues.

[0103] In the embodiment of this specification, the segmentation text prompt is the user's specified which mineral particles, cementing materials, rock fragments and pore structures need to be segmented from the sandstone image. Among them, sandstone includes nine types: quartz sandstone, feldspar quartz sandstone, rock fragments quartz sandstone, rock fragments sandstone, feldspar rock fragments sandstone, quartz rock fragments sandstone, feldspar sandstone, quartz feldspar sandstone and rock fragments quartz sandstone. The segmentation targets in sandstone include two main minerals, quartz and feldspar, one rock fragment, and three, as well as five main cementing materials, calcareous, siliceous, iron, glauconite and clay. The pore structure includes three types of primary pores, intergranular solution pores and intragranular solution pores. The diagenesis of sandstone includes sedimentary diagenesis phenomena such as compaction, pressure solution, cementation, replacement, recrystallization, and dissolution. The embodiment of this specification uses the type of sandstone segmentation target as a segmentation text prompt to guide the RockSAM model to segment the corresponding type of sandstone microscopic image.

[0104] The segmentation space prompt is a user-specified segmentation area, such as a segmentation point or a segmentation border, which guides the RockSAM model to segment targets such as specified types of mineral particles, rock fragments, cements, and pore structures in the sandstone microscopic image.

[0105] It should be noted that, in actual application, the segmented text prompt may include multiple types, and the embodiments of this specification do not limit this.

[0106] According to one embodiment of this specification, Figure 3 As shown, the steps of training the image segmentation model include:

[0107] Step 301: inputting a training image in a training image data set into an image encoder for encoding to obtain a training image embedding vector corresponding to the training image;

[0108] Step 302: input the segmented text prompt labels corresponding to the training images in the training image data set into the text prompt encoder for calculation, first generate a default semantic prompt embedding corresponding to the segmented text prompt label, and then perform weighted calculation on the default semantic prompt embedding through semantic prompt learning to obtain a training semantic prompt embedding;

[0109] Step 303: inputting the segmentation spatial cue labels corresponding to the training images in the training image data set into the spatial cue encoder for calculation, first generating a default spatial cue embedding corresponding to the segmentation spatial cue label, and then performing weighted calculation on the default spatial cue embedding through spatial cue learning to obtain a training spatial cue embedding;

[0110] Step 304: input the training semantic hint embedding and the training spatial hint embedding into the semantic spatial hint fuser for calculation to obtain the training semantic spatial fusion hint embedding;

[0111] Step 305: Input the training image embedding vector, segmentation text prompt label, segmentation space prompt label and training semantic space fusion prompt embedding corresponding to each training image into the mask decoder for decoding to obtain the segmentation result corresponding to each training image, calculate the loss value according to the segmentation result and the actual result, and when the loss value does not meet the requirements, adjust the parameters of the image encoder, text prompt encoder, space prompt encoder, semantic space prompt fusion and mask decoder, and repeat the step of inputting the training image in the training image data set into the image encoder for encoding to obtain the training image embedding vector corresponding to the training image, until the loss value meets the requirements and the trained image segmentation model is obtained.

[0112] In the embodiments of this specification, Figure 4 As shown, each attention module of the image encoder includes two adapters, the first adapter is placed after the multi-head attention layer and before the residual connection, the second adapter is placed on the residual path of the multi-layer perceptron layer after the multi-head attention layer, and the scaling factor is used after the second adapter for scaling embedding;

[0113] Inserting the adapter at a specific location ensures that the adapter can effectively interact with different parts of the model. In SAM’s image encoder, the adapter is placed after the multi-head attention and before the residual connection to ensure that the adapter can process the embedding after the attention mechanism.

[0114] like Figure 4 As shown, the adapter consists of three parts: upper projection, ReLU activation function and lower projection;

[0115] The adapter's processing steps include:

[0116] The input data is down-projected to reduce its dimension, the reduced dimension result is activated by the ReLU activation function, and then the activated result is up-projected to restore its original dimension.

[0117] Specifically, the processing steps of the image encoder with the adapter added in the embodiment of this specification can be summarized as follows:

[0118] First, the input image is normalized through the normalization layer, and then passed through the multi-head attention layer. The result of the multi-head attention layer is then input into the first adapter for the above processing. The processing result is then residually connected with the input, and the result of the residual connection is normalized and calculated by the multi-layer perceptron layer. The result of the residual connection is input into the second adapter for the above calculation. The calculation result of the second adapter is then scaled and embedded using the scaling factor. The scaled embedded result and the output of the multi-layer perceptron layer are then residually connected to obtain the image embedding vector.

[0119] It should be noted that during the training of the model, the multi-head attention layer remains frozen, while the adapter is activated, so that only a small number of adapter module parameters need to be updated, without updating all the parameters of the entire image encoder to make the model adaptable to the sandstone microscopic image segmentation task.

[0120] In order to solve the problem of limited search space in spatial cue learning of SAM, and the problem of insufficient background knowledge learning of text cue encoder. The embodiment of this specification provides a new cue learning method that combines space and semantics. It optimizes the spatial cues and semantic cues in the embedding space through a small amount of sandstone microscopic image data, and selectively uses the knowledge encoded in the cue encoder to finally learn an effective semantic-spatial fusion cue embedding to guide the mask decoder to predict the segmentation mask. Specifically, the embodiment of this specification adds a semantic spatial cue fuser component between the cue encoder and the mask decoder, which fuses the spatial information and semantic information with each other to jointly learn effective spatial and semantic cues for SAM.

[0121] Detailed principle: Figure 5 As shown, first, for semantic cue learning, the embodiment of this specification optimizes semantic cues directly in the embedding space, and selectively uses the knowledge encoded in the pre-trained text cue encoder for learning. It uses learnable weights to weight the default semantic cue embedding encoded by the text cue encoder, and fuses the weighted embedding with a set of learnable semantic cue embeddings to generate new semantic cues.

[0122] Semantic cue learning has two excellent features:

[0123] ① Semantic cue learning is efficient because its optimization only involves the embeddings encoded by the text cue encoder, not the text cue encoder itself;

[0124] ② The learned weights for semantic cue learning are able to selectively utilize the semantic knowledge in the default semantic cue embedding and the learnable semantic cue embedding, which helps to capture complementary knowledge, i.e., the former (default semantic cue embedding) is encoded by the original text cue encoder, pre-trained on large-scale data, and captures general semantic knowledge, while the latter (learnable semantic cue embedding) is optimized and learned from downstream data (i.e., sandstone microscopic image data) and largely captures the semantic knowledge in the sandstone microscopic image segmentation task.

[0125] Secondly, for spatial cue learning, the embodiment of this specification optimizes spatial cues directly on the high-dimensional embedding space and selectively utilizes the knowledge encoded in the pre-trained spatial cue encoder. It uses learnable weights to weight the default spatial cue embedding encoded by the spatial cue encoder and fuses the weighted embedding with a set of learnable spatial cue embeddings to obtain new spatial cues.

[0126] Spatial cue learning also has two excellent features:

[0127] ① It broadens the original limited search space by directly optimizing the spatial cues on a high-dimensional embedding space, which has a larger search space (e.g., 512 dimensions) than the 2D coordinate space;

[0128] ②Similar to semantic cue learning, spatial cue learning can also selectively utilize the knowledge encoded in the pre-trained spatial cue encoder.

[0129] In the last step, the embodiment of this specification uses a semantic spatial cue fuser to fuse the new semantic cues generated by semantic cue learning and the new spatial cues generated by spatial cue learning to obtain a semantic spatial fusion cue embedding, aiming to utilize the synergistic fusion of spatial information and semantic information to better prompt SAM.

[0130] Because the prompt is crucial to the segmentation effect of SAM. To this end, the embodiment of this specification provides a high-level prompt, such as Figure 6 As shown in Figure 3, the high-level cue fuses semantic cue information and spatial cue information. Through projection and reshaping operations, a series of weight maps are generated from the semantic-spatial fusion cue embedding, and then the high-level cue output is obtained.

[0131] Then, the embodiment of this specification embeds this high-level hint into the adapter module, specifically, the high-level hint is placed between the lower projection and the ReLU activation function of the adapter. Finally, a high-level hint adapter that integrates semantic and spatial hints is obtained.

[0132] This high-level cue adapter is then inserted into the attention block of the mask decoder of SAM together with the previous adapter, resulting in a high-level cue-guided mask decoder.

[0133] like Figure 7 As shown, the embodiment of the present specification deploys three adapters in each attention block of the mask decoder of SAM. The first adapter is the high-level prompt adapter mentioned above, which is placed between two residual connections and is mainly used to integrate semantic and spatial prompt embeddings; the second adapter is the same as the adapter in the image encoder, and is placed on the residual path of the multi-layer perceptron after the multi-head attention layer. The scaling factor is also used after the adapter to scale the embedding. The third adapter is deployed after the residual connection of the cross-attention of the image embedding to the prompt. After the adapter, another residual connection and layer normalization are connected to output the segmentation mask.

[0134] This specification embodiment Figure 3 The specific embodiment of the training process shown is as follows:

[0135] Prepare sandstone microscopic images: The training samples in the embodiments of this specification include a total of 9 types of sandstones, namely quartz sandstone, feldspar quartz sandstone, lithic quartz sandstone, lithic sandstone, feldspar lithic sandstone, quartz lithic sandstone, feldspar sandstone, quartz feldspar sandstone and lithic quartz sandstone. Sandstone contains three main mineral particles of quartz, feldspar and lithic, as well as five main cements of calcareous, siliceous, ferruginous, glauconite and clay. The pore structure includes three types of primary pores, intergranular solution pores and intragranular solution pores. The diagenesis of sandstone includes sedimentary diagenetic phenomena such as compaction, pressure solution, cementation, replacement, recrystallization and dissolution.

[0136] Because single polarized light cannot accurately determine the edge of mineral particles, and the extinction of mineral particles at different orthogonal polarization angles makes the edge of mineral particles more obvious and easier to segment. For this reason, the embodiment of this specification combines single polarized light and orthogonal polarized light sequence images to perform sandstone microscopic image segmentation. Each sandstone thin section has a standard thin section thickness of 0.03mm and is filled with blue gum to make a cast thin section. After selecting a representative field of view, the thin section is placed under a polarizing microscope and a single polarized light photo and 7 orthogonal polarized light photos are taken. Among them, the single polarized light photo was taken at the 0° position, and the other 7 orthogonal polarized light photos were taken at 0°, 15°, 30°, 45°, 60°, 75°, and 90° positions respectively. Each photo is 4908×3264 pixels, the image pixels are in RGB format, and are uniformly saved in JPG format.

[0137] Sample labeling: The labeling process is completed by rock and mineral identification experts according to the thin section identification industry standards using the browser-based interactive segmentation tool provided by SAM. Segmentation prompts are provided to SAM in an interactive manner by left-clicking the mouse to select foreground points, right-clicking the mouse to select background points, and drawing borders with the mouse. SAM is guided to segment the mineral particles, rock fragments, cements and pore structures in the image. At the same time, each mineral particle, rock fragment, cement and pore structure is labeled with a category, and the labeled mask image and category labels are then exported. This is a semi-automatic labeling method that greatly improves the labeling quality and efficiency.

[0138] Divide the data set: The original sandstone microscopic images and the corresponding annotated mask images and category labels are combined into training samples, and then the training samples are divided into training set, test set and validation set in a ratio of 8:1:1.

[0139] Image preprocessing: First process the training set:

[0140] In the first step, in order to reduce the amount of model calculation, the original sandstone microscopic image and the mask image are scaled to a unified size, that is, the original image 4908×3264 pixels is scaled to 2454×1632 pixels.

[0141] The second step is to normalize the image, that is, convert the pixel values ​​to the range of [0, 1], in order to maintain the stability of the numerical range, reduce the risk of model gradient disappearance, and improve training efficiency.

[0142] The third step is to crop the image, crop out the foreground area in the original microscopic image and the mask image (referring to the mineral particles, rock fragments, cement and pore structure marked in the image, etc.), and remove the background area (referring to the area without segmented objects), and then update the training set, in which the original microscopic image only contains the segmented objects, and the mask image only contains the annotations corresponding to the segmented objects. The purpose of the cropping operation is to make this new training set can be used to train our model, so that the model can focus more on the features of the segmented objects and reduce the interference of the background.

[0143] The fourth step is to adjust the pixel spacing of the original image and the mask image to the specified value (1.5, 1.5) in order to maintain the consistency of the image data.

[0144] In the fifth step, the original image and the mask image are randomly flipped in the X-axis and Y-axis directions with a probability of 15%, and then randomly rotated 90 degrees with a probability of 15%, up to 3 times. Finally, an offset is randomly added to the image with a probability of 50%. The offset size is 0.1. The purpose is to increase the diversity of image data, thereby improving the generalization ability and robustness of the model.

[0145] It should be noted that the preprocessing steps of the validation set are the same as those of the training set, but only include the first four steps, without the fifth step. No special processing is done on the test set.

[0146] Model training, validation and testing:

[0147] First, set some basic parameters for model training: including the number of epochs, the number of batches, setting the Adam optimizer to update model parameters, setting the weights of the learning rate and L2 regularization term, setting the initial evaluation score, and setting the learning rate. The number of semantic cues is set to the number of downstream task classes, the number of spatial cues is set to 100, and the embedding vector dimension is set to 512. The evaluation indicator uses the DiceCELoss (Dice Cross-Entropy Loss) loss function, which combines the Dice coefficient and the cross entropy loss. The Dice coefficient is used to measure the overlap between the predicted results and the true labels, while the cross entropy loss is used to measure the difference between the distribution of the predicted results and the distribution of the true labels.

[0148] Then, in the training process of the model, the embodiment of this specification adopts two prompt combination settings:

[0149] (1) A combination of a point cue (i.e., spatial cue) and a text cue (i.e., semantic cue) indicating the names of mineral grains, rock fragments, cements, and pore structures.

[0150] (2) A combination of a rectangular bounding box cue (i.e., spatial cue) and a text cue (i.e., semantic cue) indicating the names of mineral grains, rock fragments, cements, and pore structures. Points and rectangular bounding boxes are both sparse cues, where the point cue embedding consists of the sum of the position encoding of the point's coordinates and the learned embedding indicating whether the point belongs to a foreground point or a background point. The rectangular bounding box cue consists of two embedding vectors:

[0151] ① An embedding vector obtained by adding the position encoding of the upper left corner of the bounding box to the learned embedding representing "upper left corner";

[0152] ② Another embedding vector obtained by adding the position encoding of the lower right corner of the bounding box to the learned embedding representing “lower right corner”.

[0153] Before starting training, first load the RockSAM network, then, given an input image Name of a set of segmented objects (mineral grains, cements, rock fragments, and pore structures) As the default semantic hint (that is, the segmented text hint label in step 302) and a series of points As the default spatial hint (that is, the segmentation spatial hint label in step 303), H represents the height of the image, W represents the width of the image, 3 represents the RGB color channel values ​​of each pixel, C represents the number of semantic hints, and N represents the number of spatial hints. In the training phase, the default semantic and spatial hints are provided in the annotation phase, where each object in the image has a name and a bounding box as a hint.

[0154] Then, through the pre-trained text prompt encoder Encoder T Convert the default semantic hints to default semantic hint embeddings (vectors): Through the pre-trained spatial hint encoder Encoder S Convert default spatial cues to default spatial cues embeddings (vectors):

[0155] This is the process of initializing the segmented text hint label and the segmented space hint label. The method for generating the initial semantic hint embedding in step 222 is the same as the method for generating the default semantic hint embedding, and the method for generating the initial semantic hint embedding in step 223 is the same as the method for generating the default space hint embedding.

[0156] At the same time, the input image x I Through the image encoder Encoder I Encoded as image embedding vector: z I =Encoder I (x I ). First, the sandstone microscopic image after the image preprocessing step is reduced in dimension through the down-projection operation of the adapter in the image encoder, and then the nonlinear operation is introduced through the ReLU activation function, so that the image encoder can learn more complex features and mapping relationships. Then, the image embedding is mapped to the original dimension through the up-projection operation, and finally the image embedding vector z is output. I The purpose of adding adapters to the image encoder is to increase the flexibility and adaptability of the model. The adapter allows adapting to new tasks by training only a small number of additional parameters while keeping most of the image encoder parameters unchanged. This helps reduce the number of parameters that need to be trained, thereby saving computing resources and time.

[0157] Next, let’s start with the specific implementation process of semantic prompt learning (SemPL):

[0158] First, let represents C learnable semantic embedding vectors, where D represents the vector dimension, and represents C learnable weights, where Then, the training semantic hint embedding is obtained by weighting the default semantic hint embedding By applying W T Embedding of default semantic hints Weigh and add the weighted embedding vector to the learnable semantic embedding vector Z T The fusion is obtained as shown in formula (1):

[0159]

[0160] in, represents training semantic cue embedding, represents the cth weight, represents C learnable weights, represents the cth semantic embedding vector, represents C learnable semantic embedding vectors, Represents the default semantic hint embedding.

[0161] Among them, C are learnable semantic embedding vectors It starts with the default semantic cue embedding and continuously learns the semantic embedding vector during training by fusing the original text cue encoder and the knowledge in the downstream sandstone microscopic images.

[0162] The calculation method for training semantic cue embedding is the same as the calculation method for target semantic cue embedding in step 2022.

[0163] It should be noted that after obtaining the training semantic hint embedding, it can be decoded with the input image embedding vector to obtain the predicted segmentation result. I , default space hint embed And training semantic cue embeddings Input into the mask decoder Decoder, RockSAM can predict a set of segmentation masks using formula (2):

[0164]

[0165] Where M represents the predicted binary segmentation mask (i.e., segmentation result), S represents the confidence score corresponding to each segmentation mask, and X I represents the input sandstone microscopic image training set, y I Represents the embedding vector of the mask image corresponding to the input image. Then, formula (3) is used to continuously optimize the DiceCELoss segmentation loss function. Finally, RockSAM is able to learn the best set of training semantic cue embeddings for the sandstone microscopic image segmentation task. In this process, only the learnable semantic embedding vector Z is updated T and learnable weights W T Deoptimization All other modules in the text prompt encoder remain frozen.

[0166] It should be noted that the method shown in formulas (2) and (3) is only an implementable method. The preferred iterative method of the embodiment of this specification is as described in step 305, when the loss value does not meet the requirements, the parameters of the image encoder, text prompt encoder, spatial prompt encoder, semantic spatial prompt fuser and mask decoder are adjusted simultaneously and then iterative training is performed.

[0167] Then, the input point cues and bounding box cues are encoded into default spatial cue embeddings by the spatial cue encoder. The default spatial cue embeddings are then weighted by spatial cue learning (SpaPL) to generate training spatial cue embeddings.

[0168] The following is the specific implementation process of Spatial Prompt Learning (SpaPL):

[0169] make represents N learnable spatial embedding vectors, where D represents the dimension of the embedding vector, and Represents N learnable weights, where Training spatial cue embeddings Formula (4) can be obtained by applying W S Embed default space hints Weigh and combine the weighted embedding with the learnable spatial embedding vector Z S Combine to get:

[0170]

[0171] in, represents the training spatial cue embedding, represents the nth weight, represents N learnable weights, represents the nth spatial embedding vector, represents N learnable spatial embedding vectors, Represents the default space hint embedding.

[0172] Among them, N learnable spatial embedding vectors It starts with the default spatial cue embedding and continuously learns the spatial embedding vector during training by fusing the original spatial cue encoder and the knowledge in the downstream sandstone microscopic images.

[0173] The calculation method of the training space cue embedding is the same as the calculation method of the target space cue embedding in step 2024.

[0174] It should be noted that after obtaining the training space hint embedding, it can also be decoded with the input image embedding vector to obtain the predicted segmentation result. For example, an input image embedded vector z encoded by an image encoder is decoded with I , the default semantic hint embedding and train spatial cue embeddings Input into the mask decoder Decoder, RockSAM can predict a set of segmentation masks using formula (5):

[0175]

[0176] Where M represents the predicted binary segmentation mask, S represents the confidence score corresponding to each segmentation mask, and X I represents the input sandstone microscopic image training set, y I Represents the embedding vector of the mask image corresponding to the input image. Then, formula (6) is used to continuously optimize the training space hint embedding through the DiceCELoss segmentation loss function. Finally, RockSAM is able to learn the best set of training space cue embeddings for the sandstone microscopic image segmentation task. In this process, only the learnable spatial embedding vector Z is updated S and learnable weights W S Deoptimization All other modules in the spatial cue encoder remain frozen.

[0177] It should be noted that the method shown in formulas (5) and (6) is only an implementable method. The preferred iterative method of the embodiment of this specification is as described in step 305, when the loss value does not meet the requirements, the parameters of the image encoder, text prompt encoder, spatial prompt encoder, semantic spatial prompt fuser and mask decoder are adjusted simultaneously and then iterative training is performed.

[0178] Next, the training semantic cue embedding generated by semantic cue learning and the training spatial cue embedding generated by spatial cue learning are fused in the semantic spatial cue fuser to obtain the training semantic spatial fused cue embedding.

[0179] The following is to train the semantic space fusion hint embedding Z SSPL The specific implementation process:

[0180] The training semantic hint from formula (1) is embedded into formula (7) and the training space hint embedding of formula (5) Fusion is performed to obtain the training semantic space prompt fusion embedding Z SSPL , the implementation process is as follows:

[0181]

[0182] Among them, Z SSPL represents the training semantic space fusion hint embedding, Dense represents the fully connected layer, and D represents the target dimension of the fully connected layer conversion. Indicates that and The two vectors are concatenated into one vector. Then, they are activated by an activation function ReLU, and the final output is Z SSPL .

[0183] It should be noted that the training semantic space prompt fusion embedding Z SSPL After that, it can also be sent to the mask decoder for decoding to obtain the predicted segmentation result. For example, using formula (8) to convert Z SSPL This is input into the mask decoder to return the predicted segmentation mask M.

[0184] {M,S}=Decoder(z I |Z SSPL ) (8)

[0185] Where M represents the predicted binary segmentation mask (i.e., segmentation result), S represents the confidence score corresponding to each segmentation mask,

[0186] It should be noted that the method shown in formula (8) is only one possible implementation method and will not be described in detail here.

[0187] Next, the mask decoder integrated with the advanced hint adapter predicts the segmentation mask in the sandstone microscopic image. The following is the specific implementation process:

[0188] The mask decoder receives Z SSPL Then, as shown in formula (9), the advanced prompt adapter is used to integrate the input segmented text prompt label Split space tips label Then these two prompts are passed through the Norm normalization layer, and then down-projected to obtain Z down Then, as shown in formula (10), the training semantic space fusion cue embedding Z output by the semantic space cue fuser is used SSPL The weights W applied to the high-level cue adapter are generated through the MLP multi-layer perceptron and reshape operation SSPL The specific implementation process is as follows:

[0189]

[0190] W SSPL =Re(MLP(Z SSPL )) (10)

[0191] Where Re represents the reshaping operation and MLP represents the multi-layer perceptron.

[0192] Next, the generated weight W is SSPL and Z down Perform matrix multiplication, then normalize through the Norm layer, activate through the ReLU function, and then restore its original dimension through the Up projection operation to output Z up , through the above steps, the semantic space fusion hint is integrated into the adapter. Finally, the decoder predicts the segmentation mask based on the output of the adapter, as shown in formula (12).

[0193]

[0194] {M,S}=Decoder(Z I |Z up ) (12)

[0195] Among them, M is the predicted segmentation mask, S is the corresponding confidence score of the predicted segmentation mask, Represents matrix multiplication.

[0196] Advanced hints help adjust parameters according to hint information and adapt to different modalities and downstream tasks more flexibly.

[0197] It should be noted that the main purpose of using projection and reshaping operations in the adapter is to achieve effective parameter sharing and reduce the number of parameters. Through the projection operation, the adapter can compress the given embedding to a lower dimension, which helps to reduce the number of parameters and improve computational efficiency. The reshaping operation changes the shape of the data without changing the total amount of data. This operation does not change the dimension of the data (that is, the total number of data), but only changes the layout of the data in the network. The purpose of the reshaping operation is to adapt the shape requirements between different network layers.

[0198] In each epoch, batch training is performed according to the batch number, and then the loss between the predicted value and the true value in the entire epoch is calculated. Then the effect of the model is evaluated on the validation set. When the evaluation score reaches the best score during validation, the best model checkpoint at this moment is saved. Finally, the best model checkpoint is loaded to apply the model on the test set to complete the actual sandstone microscopic image segmentation task.

[0199] After obtaining the segmentation result, the loss value is calculated according to the segmentation result and the actual result. The loss function may be a DiceCELoss (Dice Cross-Entropy Loss) loss function. When the loss value does not meet the requirements, the parameters of the image encoder, text prompt encoder, spatial prompt encoder, semantic spatial prompt fuser and mask decoder are adjusted.

[0200] Specifically, adjusting the parameters of the image encoder, the text hint encoder, the space hint encoder, the semantic space hint fuser and the mask decoder further includes:

[0201] adjusting parameters of two adapters in the image encoder;

[0202] Adjust the weights W in the textual prompt encoder T And the learnable semantic embedding vector Z T ;

[0203] Adjust the weights W in the spatial cue encoder S and a learnable spatial embedding vector Z S ;

[0204] Adjusting the parameters of the fully connected layer Dense in the semantic space hint fuser;

[0205] Parameters of the advanced hint adapter in the mask decoder are adjusted.

[0206] Then, the step of inputting the training image in the training image data set into the image encoder for encoding to obtain the training image embedding vector corresponding to the training image is repeatedly performed until the loss value meets the requirement, thereby obtaining the trained image segmentation model.

[0207] In some other embodiments of the present specification, the image encoder, text hint encoder, space hint encoder, semantic space hint fuser and mask decoder can also be trained separately, for example, the text hint encoder is trained by formulas (1)-(3), the space hint encoder is trained by formulas (4)-(6), the semantic space hint fuser is trained by formulas (7)-(8), and the mask decoder is trained by formulas (9)-(12). Finally, a trained RockSAM model (image segmentation model) is obtained.

[0208] When using the trained RockSAM model to segment the sandstone microscopic image to be segmented:

[0209] Firstly, the sandstone microscopic image is encoded using the trained image encoder to obtain the image embedding vector;

[0210] Then, the segmented text prompts are encoded using the trained text prompt encoder to generate the initial semantic prompt embedding Then based on the trained weights and semantic embedding vector Embedding the initial semantic hint Using formula (1) to calculate, we get the target semantic hint embedding

[0211] Then, the trained spatial cue encoder is used to encode the segmented spatial cues to generate the initial spatial cue embedding Then based on the trained weights and the spatial embedding vector Embedding of initial spatial hints Using formula (4) to calculate, we get the target space hint embedding

[0212] Then, the trained semantic space cue fuser is used to embed the target semantic cue based on the parameters of the trained fully connected layer Dense. and target space cue embedding Using formula (7) to calculate, we get the semantic space fusion hint embedding Z SSPL ;

[0213] Then, the trained mask decoder is used to embed the cue in the semantic space based on the parameters of the trained high-level cue adapter. SSPL The weight W is calculated using formula (10) SSPL , and then the segmentation text prompt and the segmentation space prompt are calculated using formula (9). Then, the predicted segmentation mask corresponding to the sandstone microscopic image to be segmented and the confidence score corresponding to the predicted segmentation mask are obtained through formulas (10)-(11), where the predicted segmentation mask can be in two states, "1" or "0", for example, state "1" indicates that the image is segmented, and state "0" indicates that the image is not segmented.

[0214] The relationship between the segmentation mask and the segmentation result is common knowledge in the art and will not be described in detail here.

[0215] Based on the same inventive concept, the present specification also provides a sandstone microscopic image segmentation device, such as Figure 8 As shown, including:

[0216] The data acquisition unit 801 is used to acquire the sandstone microscopic image to be segmented, and the segmentation text prompt and segmentation space prompt corresponding to the sandstone microscopic image;

[0217] The segmentation unit 802 is used to input the sandstone microscopic image into a pre-trained image segmentation model for calculation, wherein the image segmentation model includes an image encoder, a text prompt encoder, a spatial prompt encoder, a semantic spatial prompt fuser and a mask decoder, wherein the image encoder encodes the sandstone microscopic image to generate an image embedding vector; the text prompt encoder encodes the segmentation text prompt to generate an initial semantic prompt embedding, and performs weighted calculation on the initial semantic prompt embedding through semantic prompt learning to obtain a target semantic prompt embedding; the spatial prompt encoder encodes the segmentation spatial prompt to generate an initial spatial prompt embedding, and performs weighted calculation on the initial spatial prompt embedding through spatial prompt learning to obtain a target spatial prompt embedding; the semantic spatial prompt fuser fuses the target semantic prompt embedding and the target spatial prompt embedding to obtain a semantic spatial fused prompt embedding; the mask decoder decodes the image embedding vector, the segmentation text prompt, the segmentation spatial prompt and the semantic spatial fused prompt embedding to obtain a segmentation result of the sandstone microscopic image.

[0218] Since the principle of solving the problem by the above device is similar to that of the above method, the implementation of the above device can refer to the implementation of the above method, and the repeated parts will not be repeated.

[0219] like Fig. 9 The diagram is a schematic diagram of the structure of a computer device in an embodiment of the present specification. The apparatus in an embodiment of the present specification may be a computer device in an embodiment of the present specification, executing the method in the embodiment of the present specification described above.

[0220] The computer device 902 may include one or more processing devices 904, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 902 may also include any storage resources 906 for storing any type of information such as code, settings, data, etc. Without limitation, for example, the storage resource 906 may include any one or more combinations of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any storage resource may use any technology to store information.

[0221] Further, any storage resource may provide volatile or non-volatile retention of information.

[0222] Further, any storage resource may represent a fixed or removable component of the computer device 902. In one case, when the processing device 904 executes the associated instructions stored in any storage resource or combination of storage resources, the computer device 902 may perform any operation of the associated instructions. The computer device 902 also includes one or more drive systems 908 for interacting with any storage resource, such as a hard disk drive system, an optical disk drive system, etc.

[0223] The computer device 902 may also include an input / output module 910 (I / O) for receiving various inputs (via input devices 912) and for providing various outputs (via output devices 914). A specific output mechanism may include a presentation device 916 and an associated graphical user interface (GUI) 918. In other embodiments, the input / output module 910 (I / O), the input device 912, and the output device 914 may not be included, and the computer device 902 may be used as a computer device in a network. The computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the components described above together.

[0224] The communication link 922 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0225] The embodiments of the present specification also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0226] The embodiments of the present specification also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the above method.

[0227] It should be understood that in the various embodiments of the present specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present specification.

[0228] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the embodiments of this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0229] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of this specification can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of this specification.

[0230] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0231] In the several embodiments provided in the embodiments of this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0232] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0233] In addition, each functional unit in each embodiment of the present specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0234] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0235] The embodiments of this specification use specific embodiments to illustrate the principles and implementation methods of the embodiments of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of this specification. At the same time, for those skilled in the art, according to the ideas of the embodiments of this specification, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of this specification.

Claims

1. A sandstone microscopic image segmentation method, characterized in that: The method comprises: Acquire a sandstone microscopic image to be segmented, and a segmentation text prompt and a segmentation space prompt corresponding to the sandstone microscopic image; The sandstone microscopic image is input into a pre-trained image segmentation model for calculation, wherein the image segmentation model includes an image encoder, a text prompt encoder, a spatial prompt encoder, a semantic spatial prompt fusion device and a mask decoder, wherein the image encoder encodes the sandstone microscopic image to generate an image embedding vector; the text prompt encoder encodes the segmentation text prompt to generate an initial semantic prompt embedding, and performs weighted calculation on the initial semantic prompt embedding through semantic prompt learning to obtain a target semantic prompt embedding; the spatial prompt encoder encodes the segmentation spatial prompt to generate an initial spatial prompt embedding, and performs weighted calculation on the initial spatial prompt embedding through spatial prompt learning to obtain a target spatial prompt embedding; the semantic spatial prompt fusion device fuses the target semantic prompt embedding and the target spatial prompt embedding to obtain a semantic spatial fusion prompt embedding; the mask decoder decodes the image embedding vector, the segmentation text prompt, the segmentation spatial prompt and the semantic spatial fusion prompt embedding to obtain a segmentation result of the sandstone microscopic image; The steps of training the image segmentation model include: Inputting a training image in a training image data set into an image encoder for encoding to obtain a training image embedding vector corresponding to the training image; The segmented text prompt labels corresponding to the training images in the training image data set are input into the text prompt encoder for calculation, firstly a default semantic prompt embedding corresponding to the segmented text prompt label is generated, and then the default semantic prompt embedding is weightedly calculated through semantic prompt learning to obtain a training semantic prompt embedding; Inputting the segmentation spatial cue labels corresponding to the training images in the training image data set into the spatial cue encoder for calculation, first generating a default spatial cue embedding corresponding to the segmentation spatial cue label, and then performing weighted calculation on the default spatial cue embedding through spatial cue learning to obtain a training spatial cue embedding; Inputting the training semantic hint embedding and the training spatial hint embedding into the semantic spatial hint fuser for calculation to obtain the training semantic spatial fusion hint embedding; Input the training image embedding vector, segmentation text prompt label, segmentation space prompt label and training semantic space fusion prompt embedding corresponding to each training image into the mask decoder for decoding, obtain the segmentation result corresponding to each training image, calculate the loss value according to the segmentation result and the actual result, and when the loss value does not meet the requirements, adjust the parameters of the image encoder, text prompt encoder, space prompt encoder, semantic space prompt fusion and mask decoder, and repeatedly perform the step of inputting the training image in the training image data set into the image encoder for encoding to obtain the training image embedding vector corresponding to the training image, until the loss value meets the requirements, and obtain the trained image segmentation model; Each attention module of the image encoder includes two adapters, the first adapter is placed after the multi-head attention layer and before the residual connection, the second adapter is placed on the residual path of the multi-layer perceptron layer after the multi-head attention layer, and the scaling factor is used after the second adapter for scaling embedding; The adapter consists of three parts: up-projection, ReLU activation function and down-projection; The adapter's processing steps include: The input data is down-projected to reduce its dimension, the reduced dimension result is activated by the ReLU activation function, and then the activated result is up-projected to restore its original dimension.

2. The method according to claim 1, characterized in that The default semantic hint embedding is weighted by semantic hint learning, and the formula for training semantic hint embedding is obtained as follows: in, represents training semantic cue embedding, represents the cth weight, represents C learnable weights, represents the cth semantic embedding vector, represents C learnable semantic embedding vectors, Represents the default semantic hint embedding.

3. The method according to claim 2, characterized in that The default spatial cue embedding is weighted by spatial cue learning to obtain the formula for training spatial cue embedding: in, represents the training spatial cue embedding, represents the nth weight, represents N learnable weights, represents the nth spatial embedding vector, represents N learnable spatial embedding vectors, Represents the default space hint embedding.

4. The method according to claim 3, characterized in that According to the training semantic hint embedding and the training spatial hint embedding, they are input into the semantic spatial hint fuser for calculation, and the training semantic spatial fusion hint embedding formula is obtained as follows: Among them, Z SSPL represents the training semantic space fusion hint embedding, Dense represents the fully connected layer, and D represents the target dimension of the fully connected layer conversion. Indicates that and Concatenates two vectors into one vector.

5. The method according to claim 4, characterized in that Each attention module of the mask decoder includes a high-level hint adapter, which is placed between two residual connections.

6. The method according to claim 5, characterized in that The high-level cue adapter consists of four parts: down-projection, high-level cue embedding, ReLU activation function and up-projection; The training image embedding vector, segmentation text prompt label, segmentation space prompt label and training semantic space fusion prompt embedding corresponding to each training image are input into the mask decoder for decoding, and the segmentation results corresponding to each training image are obtained, including: The segmented text prompt label and the segmented space prompt label are integrated through the normalization layer, and then the integrated result is reduced in dimension through down-projection to obtain down-projection embedding, whose formula is: Among them, Z down represents the down-projection output, Down represents the down-projection operation, and Norm represents the normalization layer. Indicates the segmentation text prompt label, Indicates the split space prompt label; The training semantic space fusion prompt embedding is calculated through the MLP multi-layer perceptron and reshaping operation to generate the weights applied to the high-level prompt adapter, which is formulated as: W SSPL =Re(MLP(Z SSPL )), where W SSPL represents weight, Re represents reshaping operation, and MLP represents multi-layer perceptron; Weight W SSPL and the lower projection output Z down Perform matrix multiplication, and then normalize the product to obtain the result of high-level prompt embedding, then activate the result of high-level prompt embedding through the ReLU activation function, and then project the activation result up to restore its original dimension to obtain the up-projection output. The formula is: Among them, Z up represents the up-projection output, Up represents the up-projection operation, and Norm represents the normalization layer. Represents matrix multiplication; Through the formula {M, S} = Decoder (Z I |Z up ) Upward projection output Z up And the corresponding training image embedding vector is decoded to obtain the segmentation result, where M is the predicted segmentation mask, S is the corresponding confidence score of the predicted segmentation mask, and Decoder represents the decoding operation.

7. The method according to claim 6, characterized in that Adjusting the parameters of the image encoder, text hint encoder, space hint encoder, semantic space hint fuser and mask decoder includes: adjusting parameters of two adapters in the image encoder; Adjust the weights W in the textual prompt encoder T ; Adjust the weights W in the spatial cue encoder S ; Adjusting the parameters of the fully connected layer Dense in the semantic space hint fuser; Parameters of the advanced hint adapter in the mask decoder are adjusted.

8. A sandstone microscopic image segmentation device, characterized in that: The device comprises: A data acquisition unit, used for acquiring a sandstone microscopic image to be segmented, and a segmentation text prompt and a segmentation space prompt corresponding to the sandstone microscopic image; The segmentation unit is used to input the sandstone microscopic image into a pre-trained image segmentation model for calculation, wherein the image segmentation model includes an image encoder, a text prompt encoder, a spatial prompt encoder, a semantic spatial prompt fusion device and a mask decoder, wherein the image encoder encodes the sandstone microscopic image to generate an image embedding vector; the text prompt encoder encodes the segmentation text prompt to generate an initial semantic prompt embedding, and performs weighted calculation on the initial semantic prompt embedding through semantic prompt learning to obtain a target semantic prompt embedding; the spatial prompt encoder encodes the segmentation spatial prompt to generate an initial spatial prompt embedding, and performs weighted calculation on the initial spatial prompt embedding through spatial prompt learning to obtain a target spatial prompt embedding; the semantic spatial prompt fusion device fuses the target semantic prompt embedding and the target spatial prompt embedding to obtain a semantic spatial fusion prompt embedding; the mask decoder decodes the image embedding vector, the segmentation text prompt, the segmentation spatial prompt and the semantic spatial fusion prompt embedding to obtain a segmentation result of the sandstone microscopic image; The steps of training the image segmentation model include: Inputting a training image in a training image data set into an image encoder for encoding to obtain a training image embedding vector corresponding to the training image; The segmented text prompt labels corresponding to the training images in the training image data set are input into the text prompt encoder for calculation, firstly a default semantic prompt embedding corresponding to the segmented text prompt label is generated, and then the default semantic prompt embedding is weightedly calculated through semantic prompt learning to obtain a training semantic prompt embedding; Inputting the segmentation spatial cue labels corresponding to the training images in the training image data set into the spatial cue encoder for calculation, first generating a default spatial cue embedding corresponding to the segmentation spatial cue label, and then performing weighted calculation on the default spatial cue embedding through spatial cue learning to obtain a training spatial cue embedding; Inputting the training semantic hint embedding and the training spatial hint embedding into the semantic spatial hint fuser for calculation to obtain the training semantic spatial fusion hint embedding; Input the training image embedding vector, segmentation text prompt label, segmentation space prompt label and training semantic space fusion prompt embedding corresponding to each training image into the mask decoder for decoding, obtain the segmentation result corresponding to each training image, calculate the loss value according to the segmentation result and the actual result, and when the loss value does not meet the requirements, adjust the parameters of the image encoder, text prompt encoder, space prompt encoder, semantic space prompt fusion and mask decoder, and repeatedly perform the step of inputting the training image in the training image data set into the image encoder for encoding to obtain the training image embedding vector corresponding to the training image, until the loss value meets the requirements, and obtain the trained image segmentation model; Each attention module of the image encoder includes two adapters, the first adapter is placed after the multi-head attention layer and before the residual connection, the second adapter is placed on the residual path of the multi-layer perceptron layer after the multi-head attention layer, and the scaling factor is used after the second adapter for scaling embedding; The adapter consists of three parts: up-projection, ReLU activation function and down-projection; The adapter's processing steps include: The input data is down-projected to reduce its dimension, the reduced dimension result is activated by the ReLU activation function, and then the activated result is up-projected to restore its original dimension.

Images