Metal microstructure image segmentation method based on text modality guidance and hierarchical fine-tuning

Abstract

The application discloses a metal microstructure image segmentation method based on text modal guidance and hierarchical fine-tuning, comprising the following steps: obtaining a scanning electron microscope image of a metal microstructure, labeling a target material structure, constructing a corresponding text description, and performing pretreatment and image enhancement; constructing a metal microstructure segmentation model based on text modal guidance; using a hierarchical fine-tuning strategy on the constructed microstructure data, and designing a multi-task joint loss function to update parameters to obtain a trained and optimized metal microstructure segmentation model; inputting the metal microstructure image and the text description into the optimized metal microstructure segmentation model to output a segmentation mask of the target microstructure. The model parameter is updated by three differentiated learning rates in different regions, which greatly reduces the calculation cost and time consumption, and at the same time, the basic generalization ability of the pre-trained model is retained.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and materials science, specifically to a method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning. Background Technology

[0002] The microstructure of materials (grains, precipitates, grain boundaries, inclusions, etc.) is a core element determining their physical properties and service behavior, and a crucial bridge connecting material preparation processes and macroscopic properties. Therefore, accurate and efficient quantitative characterization and segmentation of the microstructure of metallic materials are of great significance for revealing the intrinsic relationship between microstructure and properties, guiding new material development, and industrial quality control.

[0003] Images of the microstructure of metallic materials are often obtained through scanning electron microscopy. This type of data suffers from problems such as class imbalance, small foreground proportions, complex shapes, and strong noise interference. Traditional image segmentation methods, relying on manually set thresholds or edge detection, are ill-suited to the complexities of metallic microstructure images, resulting in poor reliability. While deep learning has provided new avenues for intelligent materials analysis in recent years, most mainstream models are designed for natural images, exhibiting poor adaptability and robustness when directly transferred to metallic microstructure images, making it difficult to meet the requirements of quantitative research.

[0004] More importantly, the weak cross-species generalization ability of existing deep learning methods in the field has become a bottleneck restricting the practical application of models. Different categories of microstructures vary greatly in morphological features, scale distribution, and imaging contrast. Current models typically perform well only when trained on microstructures of their specific category. When faced with unseen microstructures, their performance degrades sharply, failing to achieve cross-species transfer. This necessitates the construction of separate datasets and retraining for each type of microstructure in practical applications, resulting in high costs and low efficiency.

[0005] Furthermore, existing segmentation methods are mostly limited to single visual features, failing to effectively utilize prior textual information in the materials field, unable to associate visual features with semantic descriptions of the material's microstructure, and difficult to flexibly specify segmentation targets based on textual prompts, thus limiting the interpretability and interactivity of the model in complex structure identification.

[0006] On the other hand, with the development of computer vision, basic visual models have shown strong pattern recognition capabilities. However, these models have a large number of parameters. If full fine-tuning is performed directly using a limited amount of microstructure data, it will not only be computationally expensive and time-consuming, but also easily weaken the basic generalization ability of the pre-trained model, making it difficult for large models to be efficiently adapted to the task of segmenting the microstructure of metal materials. Summary of the Invention

[0007] The purpose of this invention is to provide a method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A method for segmenting metal microstructure images based on text modality guidance and hierarchical fine-tuning includes the following steps: S1. Obtain scanning electron microscope images of the metal microstructure, label the target material structure, construct the corresponding text description, and perform preprocessing and image enhancement. S2. Construct a metal microstructure segmentation model based on text modality guidance. The metal microstructure segmentation model includes a text encoder, an image encoder, a dual-path cross-modal cross adapter, a regular adapter, a scale-extended image-text spatial fusion module, and a mask decoding module, which together realize the image segmentation of metal microstructures under text modality guidance. S3. The metal microstructure segmentation model is applied to the constructed microstructure data using a hierarchical fine-tuning strategy, and a multi-task joint loss function is designed to update the parameters, resulting in a trained and optimized metal microstructure segmentation model. S4. Input the image and text description of the metal microstructure to be processed into the optimized metal microstructure segmentation model, and output the segmentation mask of the target microstructure.

[0009] Furthermore, S1 specifically includes: S1.1 Collect images of the microstructure of metallic materials under a scanning electron microscope; S1.2, Mark the microstructure regions in the scanning electron microscope image and generate the corresponding truth mask image; S1.3 Construct a text label pool and pre-set at least two description templates that conform to the material science and scanning electron microscopy characterization specifications for each type of microstructure. The description templates in the text label pool are randomly selected to provide domain-specific text modal input for training samples. S1.4. The scanning electron microscope image is converted to grayscale and then enhanced by geometric transformation and Poisson noise that simulates electron shot noise.

[0010] Furthermore, S2 specifically includes: S2.1. Use a text encoder to extract text features describing the microstructure of the input metal material; S2.2. Use an image encoder to extract the visual features of the microstructure image of the input metallic material; S2.3 Design a dual-path cross-modal cross adapter. Introduce the dual-path cross-modal cross adapter into several layers of the image encoder. Each dual-path cross-modal cross adapter includes a morphological branch and a category branch. The two branches calculate cross attention based on different text features and visual features at the same level, respectively. Configure different weight coefficients according to the network layers into which the dual-path cross-modal cross adapter is inserted, and perform weighted fusion of the attention outputs of the two branches. S2.4 Introduce regular adapters in several layers of the text encoder; S2.5 Design a scale-expanded image-text spatial fusion module to map the visual features output by the image encoder and the text features output by the text encoder to the same space and generate an image-text correlation feature map; on the image-text correlation feature map, a window attention mechanism is adopted to keep the query matrix unchanged and expand the size of the key matrix and value matrix to aggregate adjacent local features and output image-text spatial fusion features. S2.6. The mask decoding module receives the image-text spatial fusion features and outputs the segmentation mask of the microstructure image.

[0011] Furthermore, in the dual-path cross-modal adapter of S2.3: The category branch uses the category embeddings at the same level in the text encoder as the query matrix, and calculates category cross attention with the visual features at the same level as the dual-path cross-modal cross adapter to guide the visual features to extract global category semantics. The category embeddings are obtained by linear projection of the embeddings at the [EOS] position in the same level of text features.

[0012] Furthermore, in the dual-path cross-modal adapter of S2.3: The morphological branch uses the morphological embeddings at the same level in the text encoder as the query matrix, and calculates morphological cross-attention with the visual features at the same level as the dual-path cross-modal cross adapter to guide the visual features to extract local morphological semantics. The morphological embeddings are obtained by average pooling and linear projection of the remaining embeddings in the same level of text features except for the [EOS] position.

[0013] Furthermore, in the dual-path cross-modal adapter of S2.3: The weighting coefficients are preset according to the network layer where the dual-path cross-modal cross adapter is located, so that the morphological branch weight of the dual-path cross-modal cross adapter located in the middle of the image encoder is higher than the category branch weight, and the category branch weight of the dual-path cross-modal cross adapter located at the end of the image encoder is higher than the morphological branch weight.

[0014] Furthermore, the scale-expanded image-text spatial fusion module in S2.5 performs the following operations: First, the text features output by the text encoder are dimensionally adjusted. Second, the text features and visual features are mapped to the same space through the learnable mapping matrix corresponding to the text editor and the learnable mapping matrix corresponding to the image editor. Then, the cosine similarity is calculated to obtain the correlation vector between the visual features of each pixel and the microstructure text features, and the correlation vector is reorganized into a feature map according to the spatial position. Finally, the feature map is divided into region windows, and the spatial local features of adjacent windows are fused using an attention mechanism that expands the size of the key matrix and the value matrix within the window to generate image-text spatial fusion features.

[0015] Furthermore, S3 specifically includes: S3.1 Initialize the model by using the image encoder weights and text encoder weights of the DeCLIP model, which has been distilled from the visual base model DINOV2, to build an encoder framework based on ViT-B. S3.2 Construct a hierarchical fine-tuning strategy. In the shallow and middle layers of the image encoder, freeze the original parameters and fine-tune only the inserted dual-path cross-modal adapter. In the deep layers of the image encoder, update the feedforward network layer and the layer normalization layer with a first learning rate, and update the dual-path cross-modal adapter with a second learning rate, while keeping the remaining parameters frozen. The original pre-trained parameters of the text encoder are completely frozen, and only the parameters of the inserted regular adapter are updated. The parameters of the image-text spatial fusion module and the mask decoding module are completely open and updated with a third learning rate. The first learning rate is less than the second learning rate, and the second learning rate is less than the third learning rate. S3.3. Construct a multi-task joint loss function consisting of cross-entropy loss, Dice loss, and lightweight contrastive loss to optimize the metal microstructure segmentation model. Then, use the AdamW optimizer for backpropagation and weight update, and iterate until convergence to obtain the optimized metal microstructure segmentation model.

[0016] Furthermore, in S3.3, the lightweight contrast loss is calculated at the input end of the dual-path cross-modal cross adapter configured in several layers of the image encoder. The contrast loss is constructed based on the cosine similarity between the category embedding and the morphological embedding received from the corresponding level text encoder by the dual-path cross-modal cross adapter, and the average value of the contrast loss calculated by each layer is taken as the final value.

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a metal microstructure image segmentation method based on text modality guidance and hierarchical fine-tuning. It updates model parameters through three differentiated learning rates for different regions, significantly reducing computational overhead and time consumption while retaining the basic generalization ability of the pre-trained model. By introducing dual-path cross-modal cross-adaptors at multiple levels of the image encoder, and calculating category and morphological cross-attention separately, it enables material domain textual information to guide visual features hierarchically, breaking through the limitations of existing methods that rely solely on single visual features. This enhances the model's segmentation accuracy and interpretability for complex microstructures, enabling it to segment microstructures using material textual descriptions. It also effectively solves the problem of weak cross-category generalization ability in existing methods, providing technical support for the characterization and recognition of metal material microstructures and possessing promising application prospects. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention.

[0019] Figure 2 This is a general framework diagram of the image segmentation model of the present invention.

[0020] Figure 3 This is a schematic diagram of the dual-path cross-modal adapter structure of the present invention.

[0021] Figure 4 This is a schematic diagram of the window attention of the image-text spatial fusion module with scale extension of the present invention.

[0022] Figure 5 This is a diagram showing the microstructure of some of the metal materials in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, a method for segmenting metal microstructure images based on text modality guidance and hierarchical fine-tuning includes the following steps: S1. Obtain scanning electron microscope (SEM) images of the microstructure of metallic materials, label the target microstructure, construct corresponding text descriptions, and perform preprocessing and image enhancement. This step specifically includes: S1.1 Collect images of the microstructure of metallic materials under a scanning electron microscope.

[0025] S1.2, Mark the microstructure regions in the scanning electron microscope image and generate the corresponding ground truth mask image.

[0026] S1.3 Construct a text label pool, and pre-determine at least two description templates that conform to the material science and scanning electron microscopy characterization specifications for each type of microstructure. The description templates in the text label pool are randomly selected to provide domain-specific text modal input for training samples.

[0027] S1.4 The Albumentations library is used to perform grayscale processing on the scanning electron microscope images, geometric transformation is performed, and based on the electronic shot noise characteristics of scanning electron microscope imaging, Poisson distributed noise data is injected into the microstructure images to simulate the low signal-to-noise ratio scanning electron microscope imaging environment.

[0028] S2. Construct a metal microstructure segmentation model based on text modality guidance, such as Figure 2 As shown, the metal microstructure segmentation model includes a text encoder, an image encoder, a dual-path cross-modal adapter, a regular adapter, a scale-extended image-text spatial fusion module, and a mask decoding module, which together achieve text-modal guided image segmentation of metal microstructures. This step specifically includes: S2.1, A text encoder employing a 12-layer Transformer decoder block architecture extracts text features describing the microstructure of the input metallic material. Figure 2 As can be seen, in this embodiment, the microstructure of the metal material is described by textual descriptions of microstructure phases such as "globular precipitate" and "acicular phase".

[0029] The Transformer decoder block is an autoregressive generation module whose architecture consists of a masked self-attention layer (for processing the generated target sequence and preventing information leakage), a cross-attention layer (for fusing global context information from the encoder input), and a feedforward neural network. Each sub-layer is followed by residual connections and layer normalization operations.

[0030] S2.2 An image encoder using a 12-layer Transformer decoder block architecture extracts the visual features of the input metal material SEM image.

[0031] The Transformer decoder block is a bidirectional context feature extraction module. Its architecture consists of a multi-head self-attention layer (used to capture global dependencies at all positions in the input sequence in parallel) and a feedforward neural network. Each sub-layer is followed by residual connections and layer normalization operations.

[0032] S2.3 Design a dual-path cross-modal cross adapter. Introduce the dual-path cross-modal cross adapter in layers 6, 8, and 12 of the image encoder. Each dual-path cross-modal cross adapter includes a morphological branch and a category branch. The two branches calculate cross-attention based on different text features and visual features at the same level, respectively. The extraction of visual features is guided by morphological and category text semantics. Different weight coefficients are configured according to the network layers into which the dual-path cross-modal cross adapter is inserted, and the attention outputs of the two branches are weighted and fused.

[0033] Specifically, such as Figure 3 As shown, the category branch uses the category embeddings of the same level in the text encoder as the query matrix, calculates category cross attention with the adapter's level of 1 / 4 dimensionality reduction and the activated visual features, guides the visual features to extract global category semantics, and restores the enhanced features to the initial dimension. The category embedding is obtained by projecting the embeddings at the [EOS] position in the same level text features.

[0034] Among them, [EOS] (End-Of-Squence Token) is a special token that is forcibly added to the end of a text sequence in pre-trained models based on the Transformer architecture, such as CLIP.

[0035] The morphological branch uses the morphological embeddings at the same level in the text encoder as the query matrix, calculates morphological cross-attention with the dimensionality-reduced by 1 / 4 and activated visual features at the level where the dual-path cross-modal cross adapter is located, guides the visual features to extract local morphological semantics, and restores the enhanced features to the initial dimension. The morphological embeddings are obtained by average pooling and projection of the remaining embeddings outside the [EOS] position in the text features at the same level.

[0036] Based on the network layers into which the dual-channel cross-modal adapter is inserted, different weight coefficients are set for the attention output of the two branches. The morphological branch weight of the adapter located in layers 6 and 8 of the image encoder is set to 0.7, and the category branch weight is set to 0.3, guiding mid-level visual features to focus on material morphological semantics. The morphological branch weight of the adapter located in layer 12 of the image encoder is set to 0.3, and the category branch weight is set to 0.7, guiding deep visual features to focus on material category-level semantic abstraction. This allows the model to strengthen morphological perception and category discrimination capabilities at different encoding depths. The specific formulas are as follows:

[0037] Among them, F dual F is a weighted fusion feature of categorical and morphological branches. cls F represents the cross-attention output of the category branches after dimensionality increase. morphThis represents the morphological branch cross-attention output after dimensionality upgrade, α represents the weighted fusion coefficient, set to 0.3, and l represents the layer number of the image encoder.

[0038] Finally, residual connectivity coefficients are introduced at the tail of the dual-channel cross-modal adapter to adjust the fusion ratio between the enhanced features after text guidance and the original image features. The specific formula is as follows:

[0039] Among them, F out F represents the final output characteristics of the dual-channel cross-connect adapter. in This represents the initial visual features of the input dual-channel cross-modal adapter, and β is the residual connectivity coefficient, set to 0.6.

[0040] S2.4. Introduce conventional adapters in layers 6, 8, and 12 of the text encoder. The conventional adapters are commonly used in the field, such as serial bottleneck structure adapters, parallel bottleneck structure adapters, or low-rank adapters. In this example, a serial bottleneck structure adapter is used, which includes a dimension-reduced linear layer, a non-linear activation layer, and a dimension-increasing linear layer.

[0041] S2.5. Design a scale-extended image-text spatial fusion module to map the visual features output by the image encoder and the text features output by the text encoder to the same space and generate an image-text correlation feature map. In order to enhance the spatial context association between the microstructures of the material, a window attention mechanism is adopted on the image-text correlation feature map to maintain the query matrix unchanged and expand the size of the key matrix and value matrix, aggregate adjacent local features, and output image-text spatial fusion features.

[0042] Specifically, the scale-expanded image-text spatial fusion module performs the following operations: First, the features output by the text encoder are adjusted to 512 dimensions; second, text features and visual features are mapped to the same space using a learnable mapping matrix corresponding to the text editor and a learnable mapping matrix corresponding to the image editor, and the cosine similarity between the visual features of each pixel and the microstructural text features is calculated to obtain a correlation vector; then, the vectors at each spatial location are recombined to construct a feature map describing the image-text correlation; finally, multiple region windows are divided on the feature map, such as... Figure 4 As shown, while keeping the query matrix size constant within the window, the key and value matrices are expanded by 1.5 times for attention calculation. Spatial local features from adjacent windows are then fused to generate a text-image spatial fusion feature. This expansion factor aims to cover the typical interaction radius of adjacent grains, capturing the spatial structural relationships of the material's microstructure between adjacent windows. The window attention calculation formula is shown below:

[0043] Where w represents the current window, This indicates an expanded window centered on the current window, with its size increased by 1.5 times. Q w The query matrix is ​​obtained from the features of the current window. , To obtain the key matrix and value matrix from the extended window features, d k O represents the dimension of the key vector. w This represents the attention output feature, with the superscript T indicating transpose, and Softmax(·) representing the Softmax function.

[0044] S2.6. The mask decoding module receives the image-text spatial fusion features and outputs the segmentation mask of the microstructure image.

[0045] S3. Apply a hierarchical fine-tuning strategy to the constructed microstructure data using the metal microstructure segmentation model, and design a multi-task joint loss function for parameter updates to obtain the trained and optimized metal microstructure segmentation model. This step specifically includes: S3.1 Initialize the model. Use the DecLIP model, which has been distilled from the visual base model DINOV2 to enhance feature extraction capabilities, to extract its image encoder weights and text encoder weights for initialization, and build an encoder framework based on ViT-B.

[0046] Among them, DINOv2 is a self-supervised visual foundation model based on Visual Transformer (ViT). DeCLIP is an unsupervised fine-tuning framework designed specifically for open-vocabulary dense perception tasks. It effectively enhances the discriminativeness and spatial consistency of local features by decoupling CLIP's self-attention module to obtain "content" and "context" features separately, and by combining the double distillation mechanism of the visual foundation model.

[0047] S3.2 Construct a hierarchical fine-tuning strategy. In layers 1 to 10 of the image encoder, the original parameters are frozen, and only the inserted dual-path cross-modal cross adapter is fine-tuned to avoid overfitting interference from limited-scale microstructure data on basic visual features. In layers 11 to 12 of the image encoder, the feedforward network layer and the layer normalization layer are updated with a first learning rate, while the dual-path cross-modal cross adapter is updated with a second learning rate, and the remaining parameters are kept frozen. The original pre-trained parameters of the text encoder are completely frozen, and only the parameters of the inserted regular adapter are updated. The parameters of the scale-expanded image-text spatial fusion module and the mask decoding module are completely open and updated with a higher third learning rate to quickly adapt to the pixel-level classification requirements of the microstructure segmentation task. The base value of the first learning rate is set to 0.00005, the second learning rate is 0.0001, and the third learning rate is 0.001.

[0048] S3.3. Design a multi-task joint loss function consisting of cross-entropy loss, Dice loss, and lightweight contrastive loss to optimize the metal microstructure segmentation model. Use the AdamW optimizer for backpropagation and weight update, set the batch size to 4, the total number of training rounds to 100, and iterate until convergence to obtain the optimized metal microstructure segmentation model.

[0049] Specifically, the lightweight contrastive loss is calculated at the input of the dual-path cross-modal adapters configured in layers 6, 8, and 12 of the image encoder. The contrastive loss is constructed based on the cosine similarity between the category embedding and the morphological embedding received from the corresponding layer's text encoder by these dual-path cross-modal adapters. The average of the contrastive losses calculated at each layer is taken as the final value. The lightweight contrastive loss is incorporated into the multi-task joint loss function with a small weighting coefficient, ensuring that the model prioritizes segmentation accuracy optimization while using auxiliary constraints to ensure the discriminability of morphological and category embeddings, thereby improving the model's ability to distinguish different phase types and morphologies of microstructures. The specific calculation formula is as follows:

[0050] in, This represents the category embedding of the l-th layer. This represents the morphological embedding of the l-th layer. L represents the lightweight contrast loss of the l-th layer. dual For the total lightweight contrast loss, This is a smoothing term.

[0051]

[0052] Among them, L unite L represents the combined loss of multiple tasks. Dice L represents Dice loss. CE The cross-entropy loss is represented by the coefficients λ1, λ2, and λ3, which are set to 0.1, 0.7, and 0.5 in this example.

[0053] S4. Input the image and text description of the metal microstructure to be processed into the trained and optimized metal microstructure segmentation model, perform inference, and output the segmentation mask of the target microstructure, such as... Figure 5 As shown.

[0054] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning, characterized in that, Includes the following steps: S1. Obtain scanning electron microscope images of the metal microstructure, label the target material structure, construct the corresponding text description, and perform preprocessing and image enhancement. S2. Construct a metal microstructure segmentation model based on text modality guidance. The metal microstructure segmentation model includes a text encoder, an image encoder, a dual-path cross-modal cross adapter, a regular adapter, a scale-extended image-text spatial fusion module, and a mask decoding module, which together realize the image segmentation of metal microstructures under text modality guidance. S3. The metal microstructure segmentation model is applied to the constructed microstructure data using a hierarchical fine-tuning strategy, and a multi-task joint loss function is designed to update the parameters, resulting in a trained and optimized metal microstructure segmentation model. S4. Input the image and text description of the metal microstructure to be processed into the optimized metal microstructure segmentation model, and output the segmentation mask of the target microstructure.

2. The method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning according to claim 1, characterized in that, S1 specifically includes: S1.1 Collect images of the microstructure of metallic materials under a scanning electron microscope; S1.2, Mark the microstructure regions in the scanning electron microscope image and generate the corresponding truth mask image; S1.3 Construct a text label pool and pre-set at least two description templates that conform to the material science and scanning electron microscopy characterization specifications for each type of microstructure. The description templates in the text label pool are randomly selected to provide domain-specific text modal input for training samples. S1.

4. The scanning electron microscope image is converted to grayscale and then enhanced by geometric transformation and Poisson noise that simulates electron shot noise.

3. The method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning according to claim 1, characterized in that, S2 specifically includes: S2.

1. Use a text encoder to extract text features describing the microstructure of the input metal material; S2.

2. Use an image encoder to extract the visual features of the microstructure image of the input metallic material; S2.3 Design a dual-path cross-modal cross adapter. Introduce the dual-path cross-modal cross adapter into several layers of the image encoder. Each dual-path cross-modal cross adapter includes a morphological branch and a category branch. The two branches calculate cross attention based on different text features and visual features at the same level, respectively. Configure different weight coefficients according to the network layers into which the dual-path cross-modal cross adapter is inserted, and perform weighted fusion of the attention outputs of the two branches. S2.4 Introduce regular adapters in several layers of the text encoder; S2.5 Design a scale-expanded image-text spatial fusion module to map the visual features output by the image encoder and the text features output by the text encoder to the same space and generate an image-text correlation feature map; on the image-text correlation feature map, a window attention mechanism is adopted to keep the query matrix unchanged and expand the size of the key matrix and value matrix to aggregate adjacent local features and output image-text spatial fusion features. S2.

6. The mask decoding module receives the image-text spatial fusion features and outputs the segmentation mask of the microstructure image.

4. The method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning according to claim 3, characterized in that, In the dual-path cross-modal adapter of S2.3: The category branch uses the category embeddings at the same level in the text encoder as the query matrix, and calculates category cross attention with the visual features at the same level as the dual-path cross-modal cross adapter to guide the visual features to extract global category semantics. The category embeddings are obtained by linear projection of the embeddings at the [EOS] position in the same level of text features.

5. The method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning according to claim 3, characterized in that, In the dual-path cross-modal adapter of S2.3: The morphological branch uses the morphological embeddings at the same level in the text encoder as the query matrix, and calculates morphological cross-attention with the visual features at the same level as the dual-path cross-modal cross adapter to guide the visual features to extract local morphological semantics. The morphological embeddings are obtained by average pooling and linear projection of the remaining embeddings in the same level of text features except for the [EOS] position.

6. The method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning according to claim 3, characterized in that, In the dual-path cross-modal adapter of S2.3: The weighting coefficients are preset according to the network layer where the dual-path cross-modal cross adapter is located, so that the morphological branch weight of the dual-path cross-modal cross adapter located in the middle of the image encoder is higher than the category branch weight, and the category branch weight of the dual-path cross-modal cross adapter located at the end of the image encoder is higher than the morphological branch weight.

7. The method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning according to claim 3, characterized in that, The image-text spatial fusion module with scale expansion in S2.5 performs the following operations: First, the dimensions of the text features output by the text encoder are adjusted. Second, the text features and visual features are mapped to the same space through the learnable mapping matrix corresponding to the text editor and the learnable mapping matrix corresponding to the image editor. Then, the cosine similarity is calculated to obtain the correlation vector between the visual features of each pixel and the microstructure text features, and the correlation vector is reorganized into a feature map according to the spatial position. Finally, the feature map is divided into region windows, and the spatial local features of adjacent windows are fused using an attention mechanism that expands the size of the key matrix and the value matrix within the window to generate image-text spatial fusion features.

8. The method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning according to claim 1, characterized in that, S3 specifically includes: S3.1 Initialize the model by using the image encoder weights and text encoder weights of the DeCLIP model, which has been distilled from the visual base model DINOV2, to build an encoder framework based on ViT-B. S3.2 Construct a hierarchical fine-tuning strategy. In the shallow and middle layers of the image encoder, freeze the original parameters and fine-tune only the inserted dual-path cross-modal adapter. In the deep layers of the image encoder, update the feedforward network layer and the layer normalization layer with a first learning rate, and update the dual-path cross-modal adapter with a second learning rate, while keeping the remaining parameters frozen. The original pre-trained parameters of the text encoder are completely frozen, and only the parameters of the inserted regular adapter are updated. The parameters of the image-text spatial fusion module and the mask decoding module are completely open and updated with a third learning rate. The first learning rate is less than the second learning rate, and the second learning rate is less than the third learning rate. S3.

3. Construct a multi-task joint loss function consisting of cross-entropy loss, Dice loss, and lightweight contrastive loss to optimize the metal microstructure segmentation model. Then, use the AdamW optimizer for backpropagation and weight update, and iterate until convergence to obtain the optimized metal microstructure segmentation model.

9. The method for image segmentation of metal microstructures based on text modality guidance and hierarchical fine-tuning according to claim 8, characterized in that, In S3.3, the lightweight contrast loss is calculated at the input of the dual-path cross-modal cross adapter configured in several layers of the image encoder. The contrast loss is constructed based on the cosine similarity of the category embedding and the morphological embedding received from the corresponding level text encoder by the dual-path cross-modal cross adapter, and the average value of the contrast loss calculated by each layer is taken as the final value.

Images