Improved DeeplabV < 3 + >-based multi-modal rhabdolithoid fossil image semantic segmentation method

Abstract

The invention provides an improved DeeplabV < 3 + >-based multi-modal evanescent fossil image semantic segmentation method for solving the difficulty problem of evanescent fossil semantic segmentation, and mainly comprises a double-branch feature fusion coding module, a morphological prior attention module and a double-stage feature fusion module. Specifically, aiming at the problems that the distinguishing degree between a rhabdosome body and a surrounding rock is not high and the edge is fuzzy, a network is guided to pay attention to the morphological characteristics of the rhabdosome body through a morphological priori attention module; in order to capture long-distance spatial dependency and detail features at the same time, feature extraction is carried out by fusing the local feature extraction capability of CNN and the long-distance dependency capture capability of Transform; in order to solve the problems that different types of rhabdoids are small in morphological difference and lack of biological characteristics, the invention provides a double-stage multi-modal fusion method combining cross attention and channel attention, text modal meta-information of a fossil mining site is fused with an image modal, and the problem that distinguishing is difficult by only depending on an image is relieved. According to the method, the precision of semantic segmentation of the rhabdosome fossil image is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of semantic segmentation and relates to a multi-modal graptolite semantic segmentation method based on an improved DeeplabV3+. BACKGROUND

[0002] Graptolite, an extinct invertebrate animal in the marine ecosystem from Cambrian to Devonian, has important value in geological and paleontological research. The morphological analysis and classification of graptolite fossils have always been a challenge in this field. Due to the generally flat and carbonized state of the fossils and the slight morphological differences between different species, accurate identification becomes complex. Traditional graptolite classification methods rely on the experience of experts and electronic imaging techniques such as scanning electron microscope secondary imaging (SEM) and backscattered electron imaging (BSE). These methods are not only time-consuming but also require high integrity of the specimen and the identification ability of the expert. In addition, due to the limited number of fossil specimens and incomplete preservation, the application of these traditional methods is limited.

[0003] In recent years, with the development of deep learning technology, researchers have begun to explore the use of convolutional neural networks (CNN) and their variants for automatic identification and morphological understanding of graptolite images. Although some research has made some progress in classification accuracy at the genus and species levels, existing methods have limitations in practical applications. For example, the pictures used in the training of patent CN116824306B usually remove the surrounding rock information and only keep the main body of the graptolite, which limits the applicability of the model in processing fossil images containing complex backgrounds.

[0004] Semantic segmentation technology, as a method that can achieve pixel-level classification and scene understanding, is suitable for fossil image analysis that needs to distinguish between the main body and the background. However, the current general semantic segmentation algorithm cannot achieve satisfactory results in the task of graptolite semantic segmentation. This is mainly due to three main difficulties in the graptolite semantic segmentation task compared to general semantic segmentation tasks such as street scene segmentation: 1) the distinction between the main body and the background of graptolite is not high, and the edge is blurred; 2) the detailed features and overall morphological features of graptolite are crucial for a comprehensive understanding of its morphology, requiring a model with a large enough receptive field; 3) there are slight morphological differences between different species of graptolite fossils, and the original biological features may be partially lost due to damage during the formation process.

[0005] [1] Niu Z-B, Jia S-Y, Xu H-H. Automated graptolite identification at high taxonomic resolution using residual networks [J]. I science, 2024, 27(1).

[0006] [2] Niu Z-B, Xu H-H. AI-based graptolite identification improves shale gas exploration [J]. bioRxiv, 2022: 2022-01.

[0007] [3] Chen L-C, Zhu Y, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation [C]. In Proceedings of the European conference on computer vision (ECCV), 2018: 801-818. SUMMARY

[0008] The present application aims at the above-mentioned difficulties of graptolite semantic segmentation, and provides a multi-modal graptolite semantic segmentation method based on improved DeeplabV3+, so as to solve the difficulties of graptolite semantic segmentation and improve the accuracy of graptolite semantic segmentation. The technical scheme is as follows:

[0009] S1: Prepare a multi-modal graptolite fossil data set, including graptolite fossil images and their excavation site text information, and make them one-to-one correspondence;

[0010] S2: Preprocess the image data, including cropping and image enhancement, and the image enhancement includes scale random scaling and random distortion;

[0011] S2.1: The graptolite fossil image is cropped according to the size of 10 pixels above, below, left and right of each graptolite mask position, to obtain a graptolite fossil image with graptolite as the main body;

[0012] S2.2: Randomly apply scale random scaling and random distortion data enhancement methods to increase the diversity of data during training;

[0013] S3: Train the morphological prior double-branch encoding multi-modal graptolite fossil semantic segmentation network using the training set, as shown in the following formula: Figure 1 The trained neural network model is obtained, wherein the semantic segmentation network comprises a double-branch feature fusion encoding module, a morphological prior attention module and a double-stage feature fusion module.

[0014] S3.1: On the basis of the original DeeplabV3+ semantic segmentation network using ResNet34 as the backbone network, a double-branch feature fusion coding module is added, including: parallel light-weight ViT branch on the original ResNet34 branch for feature extraction; the above parallel feature extraction branch and the bilateral attention aggregation module are connected in series, and the features are fused through the bilateral attention aggregation module.

[0015] S3.1.1: The light-weight ViT branch includes the following steps: the input feature map is divided into image blocks with a size of p x p, and p takes 4, 8, 16, 32; four parallel transformer blocks are used to process the feature blocks of the above four scales respectively. Each transformer block includes a light-weight multi-head self-attention layer and a feedforward network to extract and encode features in parallel; the feature maps of different sizes are stacked and merged in the channel dimension and convolution operation is performed for integration.

[0016] S3.1.2: The bilateral attention aggregation module, as shown in Figure 2 , includes the following steps: spatial attention and channel attention are performed on the ResNet branch and the light-weight ViT branch respectively. Finally, the two features are integrated by addition operation to generate the final feature representation. This representation integrates information from different regions and channels, thereby improving the model's understanding of image content.

[0017] S3.1.2.1: For the spatial attention part, the module first calculates two groups of features obtained by convolution operation on the input feature map. Then, the two groups of features are reshaped to the dimension of C x H x W (where C represents the number of channels, H represents the height of the feature, and W represents the width of the feature). Next, transpose one of the features and perform matrix multiplication with C to generate a spatial attention matrix S through the softmax function, where each element Sij of S represents the correlation between the i-th pixel point and the j-th pixel point. Finally, combine these weights with the original feature and adjust it through a learnable parameter to get the output of spatial attention.

[0018] S3.1.2.2: The channel attention part reshapes the multiple feature maps obtained from the ViT branch into a CxHxW matrix. Then, by computing the matrix multiplication of this matrix and its transpose matrix, a channel attention map with size CxC is obtained. Each element Xij of this attention map represents the mutual dependence between the i-th channel and the j-th channel. The obtained attention map X is processed by the softmax function, and then multiplied with the original feature map A, and adjusted by a learnable parameter. This step allows the model to apply different weights on each channel, emphasizing those channels that are more important for a particular class. The processing process of the channel attention part involves reshaping the input features to form a new representation that can capture the correlation between channels.

[0019] S3.2: Add a morphological prior attention module, including: morphological prior decoder and intermediate supervision.

[0020] S3.2.1: The morphological prior decoder includes the second, third, and fourth layer feature maps extracted by ResNet and up-sampled by linear interpolation; feature compression convolution and fusion convolution are used to further process the feature maps; the final attention map is generated by the Sigmoid activation function.

[0021] S3.2.2: Intermediate supervision includes binary processing of real labels, and supervision of the binary processed labels and generated attention maps during training to guide the network to focus on the region.

[0022] S3.3: Add a two-stage multi-modal fusion method, including: using a pre-trained BERT model to vectorize the fossil excavation site text; in the first stage, using cross-attention mechanism to fuse image and text features at different levels of ResNet architecture; in the second stage, using channel attention mechanism to assign different importance to specific feature channels.

[0023] S3.3.1: The first fusion stage includes the following steps: pre-process the text and image, extract high-dimensional features of text information using BERT; use the text features as keys (K) and values (V), and the image features as queries (Q) for attention operation; achieve feature fusion through double-headed cross-attention.

[0024] S3.3.2: The second fusion stage includes the following steps: perform adaptive average pooling on the deep feature map to obtain a global visual representation V; the text embedding L is converted to a dimension matching the visual global representation through a fully connected layer; the dot product of the global visual representation V and the text embedding L,

[0025] and then the dot product result is multiplied with the original output feature map of the model to achieve dynamic adjustment of the feature channel weight.

[0026] S4: Calculate the overall loss and the binary loss of the morphology prior attention module using the cross-entropy loss function, respectively. Through multiple iterations of training, update the network parameter weights to obtain a trained penicillite fossil image semantic segmentation network.

[0027] S5: Test the semantic segmentation effect, input the penicillite fossil image to be segmented and the excavation site text of the fossil into the segmentation model to obtain the penicillite image semantic segmentation result.

[0028] The present application proposes a multi-modal penicillite semantic segmentation method based on improved DeeplabV3+, aiming to solve the challenges faced by traditional semantic segmentation techniques in processing penicillite fossil images. Compared with the prior art, the present application significantly improves the accuracy of semantic segmentation, mainly due to the following innovations: 1) By combining the global perception ability of the visual transformer and the detail capture ability of ResNet, the global context perception ability of the model for penicillite fossil image features is optimized, thereby enhancing the recognition and segmentation performance of the model for penicillite images. 2) Introduce the morphology prior attention module, which uses the morphology prior spatial attention mechanism of intermediate supervision to guide the model to focus on the key area where the penicillite body is located, significantly improving the model's ability to distinguish penicillite and surrounding rock. 3) The present application integrates the text information of the fossil excavation site in the penicillite fossil dataset, and realizes the deep integration between modalities through cross-attention mechanism and channel attention mechanism. This multi-modal fusion strategy breaks through the limitations of traditional single image modalities, effectively improving the accuracy of the model in penicillite fossil semantic segmentation. Therefore, the present application not only improves the segmentation accuracy of penicillite fossil images, but also enhances the model's recognition ability of penicillite fossils in complex geological backgrounds through multi-modal information fusion, which has important practical significance for geological research and related fields. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 Multi-modal penicillite semantic segmentation network structure diagram based on improved DeeplabV3+

[0030] Figure 2 Double attention aggregation module structure diagram

[0031] Figure 3 Comparison chart of segmentation results of penicillite fossil images by baseline method and the method of the present patent

[0032] Figure 4 Results of baseline method and the method of the present patent on the test set (the method of the present patent is represented by FossilNet)

[0033] Figure 5 Multi-modal penicillite semantic segmentation network framework based on improved DeeplabV3+ (summary diagram) DETAILED DESCRIPTION

[0034] The application will be further described below in conjunction with the accompanying drawings and specific embodiments, but the following embodiments are by no means limiting to the application.

[0035] First step, prepare the data set.

[0036] (1) Preprocess and annotate the graptolite fossil data set to obtain the training set:

[0037] The application uses an expert-annotated 113-class graptolite fossil image data set. First, the original image is one-to-one corresponding to the fossil excavation site, and multi-modal data is prepared. Second, for image data, the image is cropped according to the mask of each graptolite, leaving 10 pixels on the top, bottom, left and right of the original image, to obtain graptolite images with graptolite as the main body.

[0038] (2) Image enhancement:

[0039] In the training process, the application randomly applies two data enhancement methods, scale random scaling and random distortion, to increase the diversity of data and improve the generalization ability of the model.

[0040] a) Scale random scaling: through scale random scaling, the model can learn to adapt to graptolite images of different sizes, improving the applicability of the model. Specifically, by randomly selecting a scaling ratio in the range of 0.5 to 2.0 with a step of 0.25, the image is scaled proportionally. Such processing not only increases the diversity of image sizes in the data set, but also simulates different sizes of graptolites that may be encountered in natural environments, enabling the model to be trained at various scales, thereby improving the recognition and segmentation capabilities of different sizes of graptolites. Especially in the case of limited sample size, such multi-scale training is particularly important.

[0041] b) Random distortion: to address the image quality issues caused by changes in color and contrast of graptolite images under different lighting and environmental conditions, random distortion technology is used for processing. This technology randomly adjusts the brightness, contrast, saturation and hue of the image to increase the robustness of the model to image changes, making it easier to adapt to different shooting conditions.

[0042] Second step, build a multi-modal graptolite semantic segmentation network of DeeplabV3+.

[0043] The DeeplabV3+ network is composed of an encoder-decoder structure and has been widely used in the field of general semantic segmentation. The application modifies the DeeplabV3+ network and adds three network modules. ResNet34_vd is used as the main feature extraction network, and residual stride convolution blocks are used to achieve multi-scale downsampling of feature maps.

[0044] 2.1 Increase the dual-branch feature fusion coding module, through the multi-cut size lightweight ViT to model the long-distance of the multi-size features of the shallow backbone network features, in order to capture the global context information of the image. Including: parallel ResNet branch, lightweight ViT branch and bilateral attention aggregation module in series with the first two branches. Use lightweight backbone feature extraction network ResNet34_vd as the backbone feature extraction model, and realize multi-scale down-sampling of the feature map through the residual stride convolution block. Copy the features after the second down-sampling to the lightweight ViT module for multi-head attention modeling. Through the bilateral attention mechanism, the spatial information extracted by the ResNet branch is fused with the global information extracted by the ViT branch.

[0045] 2.1.1 The lightweight ViT branch includes the following steps:

[0046] Given the input feature map First, it is divided into image blocks of size p x p (p = 4, 8, 16, 32), and each image block is flattened into a vector Apply linear projection to each vector to get block embedding e i , denoted as:

[0047] e i = W embed ·v i +b embed

[0048] Where W embed and b embed are the weights and biases of the linear projection, respectively.

[0049] Then, four parallel transformer blocks with sizes p = 4, 8, 16, 32 are used to process features of different scales. Each transformer block includes a lightweight multi-head self-attention layer and a feed-forward network to extract and encode features in parallel.

[0050] The calculation of the multi-head self-attention layer can be represented as:

[0051]

[0052] Where Q, K, V represent the query, key and value matrices respectively, and d k is the dimension of the key.

[0053] The output of the multi-head self-attention layer is further encoded by the feed-forward network, and the output of the feed-forward neural network (FFN) of the branch is represented as:

[0054] F t = ReLU(WFFN A+b FFN )

[0055] where W FFN and b FFN are the weights and biases of the feedforward network.

[0056] The feature maps obtained by the four parallel ViT branches are stacked and merged in the channel dimension, and convolution operation is performed to integrate the features of each size patch. Finally, the attention fusion is performed with the feature map of ResNet for subsequent image processing process.

[0057] 2.1.2 ResNet branch The following steps are used: ResNet34_vd is used as the backbone network, and local features are extracted through virtual deep convolutional layers and residual blocks. The "short chain gradient backpropagation" strategy is introduced, the deep features are sent to an independent decoder, and the morphological extraction ability is optimized through intermediate supervision strategy. Global pooling and geographic information prior module are used to enhance the guidance of class key information in the feature extraction process.

[0058]

[0059] where and represent the morphological prior module loss function, the second fusion stage loss function, and the segmentation decoder loss function, respectively.

[0060] At the same time, the first fusion stage occurs at this position, and the cross attention is introduced to directly add class information guidance to each layer of the backbone feature extraction network.

[0061]

[0062] where F next represents the feature sent to the next feature extraction layer, F input represents the input feature of the feature extraction layer, W represents the feature vector after text vector encoding, Q represents a learnable matrix, and D represents the dimension of the text feature vector.

[0063] 2.1.3 The bilateral attention aggregation module includes the following steps:

[0064] In order to avoid performance degradation caused by simple addition, a bilateral attention aggregation module is used to aggregate the features extracted by ResNet and ViT. Specifically, the spatial attention part of this module can be represented by the following formula:

[0065]

[0066] where W i is the input feature F​j Features extracted through convolution, and B i and C j B is the feature generated by the input feature layer through convolution and transformation transpose. i ∈R C×(H×W) Spatial attention distinguishes and activates the original attention map by comparing it with global information of maximum confidence, and then integrates them through multiplication.

[0067] Channel attention can be represented by the following formula, where A is the feature reshaped by the input feature $X$, and A∈R. b×c×h×w .

[0068]

[0069] Channel attention enhances the complementarity and expressiveness of features through the Softmax function and maximization operation. In the above two equations, α and β are parameters learned automatically.

[0070] Finally, the two-sided features are summed element by element, and the output feature is represented by E:

[0071] E = E c +E s

[0072] 2.2 A morphological prior attention module is added. Through an intermediate-supervised morphological prior spatial attention mechanism, the model focuses on the key region where the graptolite body is located. Specifically, by using an attention-guided decoder and an intermediate-supervised strategy, the model's ability to extract graptolite morphological information is enhanced, and the distinction between the graptolite body and the surrounding rock is improved.

[0073] 2.2.1 The morphological prior decoder includes the following key steps: This module borrows the feature decoding strategy used in FCN. First, the three deepest layers of features extracted by ResNet are fed into this module. Due to differences in spatial size between feature maps, linear interpolation is used to upsample the feature maps to ensure spatial consistency in subsequent feature processing. Next, features are compressed to suppress irrelevant information in the channel dimension and reduce computational cost. The compressed feature maps are then concatenated along the channel dimension to achieve feature fusion. This fusion method allows features from different levels to complement each other, improving the model's ability to capture both global and local information.

[0074] use Let i∈{2,3,4} represent the feature map extracted from ResNet, where X4 represents the deepest feature. Feature compression and fusion are calculated as follows:

[0075]

[0076] in, denotes the fusion convolution of X3 and X2, and denotes the corresponding compressed convolution, denotes the fusion convolution of X4 and X3, and denotes the feature splicing.

[0077] Finally, the three feature maps are fused together and reorganized into the 0-1 interval by the Sigmoid function, and the features obtained by the encoder are reorganized to further improve the feature extraction effect. The module output is as follows:

[0078]

[0079] wherein, denotes the fusion convolution of X3 and X2, denotes the output convolution, and σ denotes the Sigmoid activation function.

[0080] The intermediate supervision strategy in 2.2.2 includes the following key steps: generating a binary ground truth label, binarizing the part label of the graptolite and other areas such as surrounding rock to form a distinction between foreground and background. The binary cross-entropy loss function is used to calculate the loss between the binary ground truth label and the attention map generated by the attention-guided decoder. The calculated loss value is used to supervise and optimize the parameters of the attention-guided encoder to ensure the consistency of the generated attention map with the human expert's attention area.

[0081] In the graptolite fossil semantic segmentation task with 113 types, the G i , i∈{1,…,113} represents the binary ground truth label of the i-th type of graptolite image, wherein the foreground represents a specific graptolite, and the background represents all other contents such as surrounding rock. The binary ground truth label G b is calculated as follows:

[0082]

[0083] The binary cross-entropy loss function is then used to calculate the binary loss between the binary ground truth label and the attention map generated by the attention-guided decoder. The calculated binary loss is then used to supervise the parameter update of the attention-guided encoder.

[0084] Through the integration of the morphological prior attention module, the present application can significantly improve the feature extraction capability and segmentation accuracy in the graptolite fossil image semantic segmentation task without significantly increasing the computational burden, especially when dealing with complex fossil images with similar backgrounds.

[0085] 2.3 Increase the two-stage multimodal fusion method, the meta-information of the graptolite fossils such as the excavation site is integrated into the model to guide feature extraction and pixel classification, and the accuracy of pixel classification is improved. First, the pre-trained BERT model is used to vectorize the text of the fossil excavation site; in the first stage, the cross-attention mechanism is used to fuse image and text features at different levels of the ResNet architecture; in the second stage, the channel attention mechanism is used to assign different importance to specific feature channels.

[0086] 2.3.1 The first fusion stage includes the following steps: using the BERT text encoder to extract high-dimensional features of the text information and adjusting them to a dimension suitable for fusion with image features. The image features extracted by ResNet are flattened for cross-attention fusion with text features. Through the cross-attention mechanism, the text features are taken as the key (K) and value (V), and the image features are taken as the query (Q) for operation, and the fused features F are obtained fused :

[0087] F fused =Attention(Q,X,V)

[0088] Residual connections are added after each fusion operation to facilitate gradient flow and enhance the flexibility of the module.

[0089] 2.3.2 The second fusion stage includes the following steps: adaptive average pooling is performed on the deep feature maps extracted by the double-branch feature extraction to obtain a global visual representation V. The text description is encoded by BERT to obtain an embedding representation L, which is converted to a dimension matching the visual global representation through a fully connected layer. Element multiplication is performed between the text embedding L and the visual global representation V to generate a fusion vector, and element multiplication is performed between the fusion vector and the initial output F of the model to dynamically adjust the channel weight of the feature map:

[0090] F'=F⊙(V⊙L)

[0091] The feature map F' adjusted by the channel attention is sent to the segmentation head to generate the final semantic segmentation map.

[0092] The beneficial effects of this step mainly include: the cross-attention mechanism in the first stage allows the model to dynamically adjust the importance of image features based on text information, thereby paying more attention to key parts of the image. The channel attention mechanism in the second stage further adjusts the channel weight of the model output, enhancing the recognition ability of specific types of graptolites. Experiments show that the two-stage fusion method not only improves the accuracy of segmentation, but also speeds up the model's understanding speed for different types of graptolites.

[0093] Specific operation, first calculate the mean file of all training images, subtract the mean file from the images used for training, and then input into the neural network, after forward propagation and reverse conduction, the network parameters are updated once, and after multiple iterations, the trained neural network can be obtained as the graptolite semantic segmentation network.

[0094] Third step, test the semantic segmentation effect of the system

[0095] 3.1 During testing, the graptolite fossil image to be segmented and the excavation site text of the fossil are input into the segmentation model, the segmentation model is inferred, and finally the semantic segmentation result of the graptolite fossil image is obtained.

[0096] 3.2 The results of the segmentation model and other general semantic segmentation models are compared, as shown in Figure 3 It can be found that the graptolite main body can be better distinguished using the present application, and each pixel is assigned a correct category. As shown in Figure 4 The effect of the present application on the graptolite semantic segmentation task can be seen from the evaluation index.

Claims

1. A multi-modal penicillite fossil image semantic segmentation method based on improved DeeplabV3+, characterized by, The method comprises the following steps: Step 1) preparing a multi-modal data set, including associating the graptolite fossil image with its text information of the digging site; Step 2) preprocessing the image data, including cropping and image enhancement, and the image enhancement includes random scaling of the scale and random distortion; Step 3) training a double-branch coding multi-modal graptolite fossil semantic segmentation neural network with morphological prior by using a training set, to obtain a trained neural network model, wherein the neural network model comprises a double-branch feature fusion coding module, a morphological prior attention module with intermediate supervision, and a decoding module.

2. The method of claim 1, wherein, The double-branch feature fusion coding module comprises: a parallel ResNet branch and a lightweight ViT branch; and a series double-branch feature fusion module that fuses features through a double-sided attention aggregation mechanism.

3. The method of claim 2, wherein, The lightweight ViT branch comprises the following steps: dividing the input feature map into image blocks with a size of p x p, and p is 4, 8, 16, or 32; using four parallel transformer blocks to process the feature blocks of the four scales respectively. Each transformer block comprises a lightweight multi-head self-attention layer and a feedforward network to extract and encode features in a parallel manner; and stacking and merging the feature maps of different sizes in the channel dimension and performing convolution operation integration.

4. The method of claim 2, wherein, The double-sided attention aggregation module comprises the following steps: performing spatial attention and channel attention on the ResNet branch and the lightweight ViT branch respectively. Finally, the two features are integrated by addition operation to generate the final feature representation. This representation integrates information from different regions and channels, thereby improving the model's understanding of image content.

5. The method of claim 4, wherein, For the spatial attention part, the following steps are included: first, two groups of features are calculated by performing convolution operation on the input feature map. Then, the two groups of features are reshaped to a dimension of C x H x W (where C represents the number of channels, H represents the height of the feature, and W represents the width of the feature). Next, one of the features is transposed, and the transposed feature is multiplied by C to generate a spatial attention matrix S through a softmax function, where each element Sij of S represents the correlation between the i-th pixel and the j-th pixel. Finally, the weights are combined with the original feature, and a learnable parameter is used for adjustment to obtain the output of the spatial attention.

6. The method of claim 4, wherein, The channel attention part comprises the following steps: the multiple feature maps obtained by the ViT branch are reshaped to a matrix form of C x H x W. Then, a channel attention map with a size of C x C is obtained by calculating the matrix multiplication of the matrix and its transpose matrix. Each element Xij of the attention map X represents the mutual dependence between the i-th channel and the j-th channel. The attention map X obtained by the softmax function is then multiplied by the original feature map A, and a learnable parameter is used for adjustment. This step allows the model to apply different weights on each channel, thereby emphasizing those channels that are more important for a specific class. The processing process of the channel attention part involves reshaping the input feature to form a new representation that can capture the correlation between channels.

7. The method of claim 1, wherein, The morphological prior attention module comprises a morphological prior decoder and intermediate supervision.

8. The method of claim 7, wherein, The morphological prior decoder comprises second, third and fourth layer feature maps extracted by ResNet, and is up-sampled by linear interpolation. The feature compression convolution and fusion convolution are used to further process the feature maps, and the Sigmoid activation function is used to generate the final attention map.

9. The method of claim 7, wherein, The intermediate supervision comprises binary processing of the real label, and supervision of the binary processed label and the generated attention map in the training process to guide the network to focus on the region.

10. The method of claim 1, wherein, The two-stage multi-modal fusion method comprises: using a pre-trained BERT model to vectorize the text of the fossil excavation site; in the first stage, using cross attention mechanism to fuse image and text features at different levels of ResNet architecture; in the second stage, using channel attention mechanism to give different importance to specific feature channels.

11. The method of claim 10, wherein, The first fusion stage comprises the following steps: pre-processing the text and image respectively, using BERT to extract high-dimensional features of the text information; using the text features as keys (K) and values (V), and the image features as queries (Q) for attention operation; and realizing feature fusion through cross attention mechanism.

12. The method of claim 11, wherein, The second fusion stage comprises the following steps: performing adaptive average pooling on the deep feature map to obtain a global visual representation V; converting the text embedding L into a dimension matching the visual global representation through a fully connected layer; multiplying the global visual representation V and the text embedding L, and then multiplying the multiplication result with the original output feature map of the model to realize dynamic adjustment of the feature map channel weight.

13. The method of any one of claims 1 to 12, wherein, The cross-entropy loss function is used to calculate the overall loss and the binary loss of the morphological prior attention module, and the network parameter weight is updated through multiple iterations to obtain the trained graptolite fossil image semantic segmentation network.

14. The method of any one of claims 1 to 12, wherein, When testing the semantic segmentation effect, the graptolite fossil image to be segmented and the text of the excavation site of the fossil are input into the segmentation model to obtain the semantic segmentation result.

Images