SAM-based text-guided multi-view Mars topographic feature image segmentation method

Abstract

The invention relates to the technical field of image processing and computer vision, in particular to a text-guided multi-view Mars topographic feature image segmentation method based on SAM, and the method comprises the steps: constructing and training an MTSNet model, inputting a to-be-segmented Mars image and a topographic prompt text into the trained MTSNet model, and obtaining a segmentation result; the MTSNet model comprises an MTG-SAM branch, an LTE (Long Term Evolution Network) branch, a feature fusion module, a text prompt encoder and a Mask Decoder; the MTG-SAM branch is formed by cascading a plurality of Transform Blocks, and a TCAM (Ternary Content Addressable Memory) module is introduced into each Transform Block; the text prompt encoder comprises a CLIP text encoder, a text affine layer and a prompt encoder; according to the method, the accuracy and convenience of terrain segmentation of the Mars image are effectively improved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing and computer vision, and in particular to a text-guided multi-view Mars terrain feature image segmentation method based on SAM. Background Art

[0002] Since the 1960s, humans have never stopped exploring Mars. With the continuous advancement of deep space exploration technology, Mars exploration has once again become the focus of global attention. In the process of Mars exploration, semantic segmentation technology of Mars surface terrain plays a vital role. In recent years, deep learning technology has made significant progress in fields such as autonomous driving, human-computer interaction, and medical image analysis. However, there are huge challenges in applying these mature technologies to unstructured extreme environments such as the Martian environment. The terrain on the surface of Mars is highly unstructured and lacks regular geometric shapes or texture patterns, which makes it infeasible to directly apply existing deep learning models. In addition, the small scale and imbalance of Mars data also increase the difficulty of model training. At the same time, there are huge differences in the characteristics of images collected by Mars rovers at different positions and perspectives, which further increases the difficulty of accurate semantic segmentation of terrain.

[0003] In recent years, the emergence of basic visual models has attracted widespread attention. These models, based on deep learning architectures and trained on large-scale and diverse image datasets, have achieved strong feature extraction and zero-shot generalization capabilities. This means that even for unknown scenarios, they can still adapt to the semantic knowledge of new scenarios through appropriate adjustments or transfer learning. The recently emerging SAM (Segment Anything Model) is a basic model for prompt-based segmentation tasks that can segment objects of interest based on semantic prompts given in the form of bounding boxes, points, or masks. Although SAM achieves excellent segmentation performance in general scenarios, its performance is still trapped in local optimality when applied to the specific distribution data of the Martian surface environment due to the significant difference in data distribution between the source and target domains. In addition, SAM and most of its variants are usually only able to perform semantic segmentation using prompts in the form of points, bounding boxes, etc., ignoring the huge potential of text prompts, which can significantly reduce the annotation cost and accelerate the training process. Summary of the invention

[0004] To solve the above problems, the present invention provides a SAM-based text-guided multi-view Mars terrain feature image segmentation method, including constructing and training an MTSNet model, inputting the Mars image to be segmented and the terrain prompt text into the trained MTSNet model to obtain a segmentation result; the MTSNet model includes an MTG-SAM branch, an LTEN branch, a feature fusion module, a text prompt encoder and a Mask Decoder; the MTG-SAM branch is composed of a plurality of Transformer Blocks in cascade, and a TCAM module is introduced into each Transformer Block; the text prompt encoder includes a CLIP text encoder, a text affine layer and a prompt encoder.

[0005] The training process of the MTSNet model includes the following steps:

[0006] S1. Obtain training image data and its corresponding terrain text prompt data for preprocessing to obtain preprocessed image data; the preprocessed data includes a plurality of image mask terrain text pairs, each image mask terrain text pair includes an image, a mask image and a terrain text;

[0007] S2. Initialize the MTSNet model, freeze all parameters except the TCAM module in the MTG-SAM branch, and freeze the CLIP text encoder parameters;

[0008] S3. Input the image into the MTG-SAM branch and the LTEN branch respectively to obtain the first branch feature and the second branch feature;

[0009] S4. Inputting the first branch feature and the second branch feature into a feature fusion module to obtain a fusion feature;

[0010] S5. Input the terrain text corresponding to the image into the text prompt encoder to obtain text features;

[0011] S6. Input the fused features and text features into the Mask Decoder to obtain the segmentation result; calculate the loss training model according to the segmentation result and the mask image until the model parameters converge.

[0012] Beneficial effects of the present invention:

[0013] This method can achieve the segmentation of target terrain according to the given Mars terrain text. The model has strong adaptability and good performance, and effectively improves the accuracy and convenience of Mars image terrain segmentation.

[0014] The present invention solves the problem that the traditional deep learning model cannot be effectively adapted due to the highly unstructured terrain of the Martian surface and the lack of conventional geometric shapes or texture patterns, and the problem that the SAM model has poor performance when applied to the specific distribution data of the Martian surface environment due to the difference in the distribution of the training data set (natural images on the earth) and the target data (Mars images). At the same time, it also solves the problem that SAM and most of its variants can usually only use prompts in the form of points, bounding boxes, etc. for image segmentation, resulting in high annotation costs due to the complexity of the Martian terrain when segmenting Martian images. The present invention combines text prompts and image information to achieve efficient modeling and deep feature fusion of the global and local features of the Martian terrain. Compared with existing methods, the method of the present invention performs well in experimental results and significantly improves the segmentation accuracy. In addition, by introducing text as a segmentation prompt, the method of the present invention can also effectively reduce the data annotation cost, and only a small amount of text prompts are required to achieve high-quality terrain segmentation, thereby providing a more flexible and efficient solution for Martian image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a text-guided multi-view Mars terrain feature image segmentation method based on SAM of the present invention;

[0016] Figure 2 A schematic diagram of the MTSNet model framework according to an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of a GFM module framework according to an embodiment of the present invention;

[0018] Figure 4 A schematic diagram of a TCAM module framework according to an embodiment of the present invention;

[0019] Figure 5 A schematic diagram of an LTE Block according to an embodiment of the present invention;

[0020] Figure 6 This is a simulation result diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The invention provides a text-guided multi-view Mars terrain feature image segmentation method based on SAM, comprising constructing and training an MTSNet model, inputting a Mars image to be segmented and a terrain prompt text into the trained MTSNet model, and obtaining a segmentation result; the MTSNet model comprises an MTG-SAM branch, an LTEN branch, a feature fusion module, a text prompt encoder and a Mask Decoder; the MTG-SAM branch is composed of a plurality of Transformer Blocks in cascade, and a TCAM module is introduced into each Transformer Block; the text prompt encoder comprises a CLIP text encoder, a text affine layer and a prompt encoder.

[0023] like Figure 1 As shown, the training process of the MTSNet model includes the following steps:

[0024] S1. Obtain training image data and its corresponding terrain text prompt data for preprocessing to obtain preprocessed image data; the preprocessed data includes a plurality of image-mask terrain-text pairs, each of which includes an image, a mask image and a terrain text.

[0025] Specifically, the images in the training image data contain different terrain categories, such as sand, soil, bedrock and big rock; different terrain categories are described by corresponding terrain texts. Step S1 preprocesses the training image data and its corresponding terrain text prompt data, specifically including:

[0026] S11. Perform pixel standardization on each image, including: calculating the pixel mean and pixel standard deviation of the image in the three RGB channels respectively; for each pixel value of each channel in the image, subtract the pixel value from the mean of the corresponding channel and divide it by the standard deviation of the corresponding channel.

[0027] Specifically, for an image of size 512×512, the pixel means of the R channel image, G channel image, and B channel image of the image are 123.675, 116.28, and 103.53, respectively, and the pixel standard deviations are 53.395, 57.12, and 57.375, respectively. For each pixel value in the R channel image, the pixel mean 123.675 is subtracted and then divided by the pixel standard deviation 53.395 to obtain an updated pixel value; the same processing is performed on the pixel values ​​in the G channel image and the B channel image.

[0028] S12. For each pixel-normalized image, mask processing is performed according to the corresponding terrain text prompt. A corresponding mask image is produced for each type of terrain, and N image-mask-terrain-text pairs are obtained, where N is the number of terrain types contained in the image; wherein, in each mask image, the value of the pixel corresponding to the terrain to be segmented is 1, and the values ​​of the remaining pixels are 0.

[0029] S13. The training image data is transformed to obtain enhanced data, where the transformation operations include horizontal flipping, vertical flipping, rotation, horizontal translation of 10%, vertical translation of 10%, brightness change, and random cropping.

[0030] Specifically, 50% of the images in the training image data are randomly selected to be horizontally flipped, and 50% of the images are randomly selected to be vertically flipped.

[0031] S14. Perform the operations of steps S11-S12 on each new image in the enhanced data.

[0032] S2. Initialize the MTSNet model, freeze all parameters except the TCAM module in the MTG-SAM branch, and freeze the CLIP text encoder parameters.

[0033] Specifically, the image encoder (i.e., MTG-SAM branch), hint encoder, and Mask Decoder are initialized using the pre-trained SAM ViT-Base model; the TCAM module and text affine layer are initialized using a random normal distribution, the CLIP text encoder is initialized using a pre-trained CLIP model, and the LTEN branch is initialized using a random normal distribution. Among them, all parameters in the MTG-SAM branch except the TCAM module and the CLIP text encoder parameters do not participate in model training and are therefore frozen.

[0034] Specifically, the AdamW optimization algorithm is used when training the model. Among them, the initial learning rate is 0.0001, and the weight decay coefficient is 0.01. It should be noted that the AdamW algorithm used in the embodiment of the present invention is improved on the basis of Adam, especially for the processing method of weight decay. AdamW separates the weight decay and momentum mechanisms, and directly adds the weight decay term to the update rule, which helps to control the complexity of the model more accurately and avoid overfitting. By independently adjusting the weight decay, AdamW can ensure that the adjustment of the learning rate is more stable, and even at different stages of the network, it can maintain the smoothness of parameter updates, thereby improving the generalization ability of the model.

[0035] The EarlyStopping method is used in the model training process of the embodiment of the present invention. When the accuracy of the verification set is no longer improved, or when the number of training times reaches the set maximum value, the training process automatically terminates. The verification set in the training process accounts for 20% of the total training set.

[0036] S3. Input the image into the MTG-SAM branch and the LTEN branch respectively to obtain the first branch features and the second branch features.

[0037] Specifically, the MTG-SAM branch adds a TCAM (Terrain Context Attention Adapter Module) module on the basis of the original SAM image encoder. Specifically, the TCAM module is attached to the multi-head self-attention of each Transformer Block of the original SAM image encoder. Figure 2 As shown, each Transformer Block includes a cascaded first LN layer, a multi-head attention layer, a TCAM module, a second LN layer and an MLP layer, wherein there is a residual branch between the Transformer Block input and the TCAM module output, and there is a residual branch between the second LN layer input and the MLP layer output.

[0038] Specifically, Figure 4 As shown in the figure, the TCAM module includes a channel attention layer, a spatial attention layer, and a spatial transformation layer. The channel attention layer includes parallel global maximum pooling and global average pooling, both of which share an MLP with a compression rate of 0.25, and the outputs of both are spliced ​​through the Sigmoid activation function layer after being spliced ​​through the MLP. The spatial attention layer includes parallel global maximum pooling and global average pooling, and the outputs of both are spliced ​​along the channel dimension in the channel splicing layer, and then pass through a 1×1 convolution layer and a Sigmoid activation function layer; the spatial transformation layer includes a downsampling layer, a ReLU activation function layer, and an upsampling layer, where the downsampling layer uses a 3×3 convolution and the upsampling layer uses a 4×4 device convolution.

[0039] In the present invention, the input feature map F of the TCAM module 0 The size is C×M×M, where M×M represents the scale of the input feature map and C represents the dimension of the input feature map. 0 After the global maximum pooling and global average pooling in parallel in the channel attention layer, two feature maps F of size C×1 are obtained. 1 、F 1 ', and then pass through MLP to obtain two feature maps F of size C×1 2 、F 2 ';Feature map F 2 、F 2'After splicing, pass through the Sigmoid activation function layer, and connect the Sigmoid activation function layer output to the input feature map F 0 Multiply to get the feature map F 3 . Feature map F 3 After the global maximum pooling and global average pooling in parallel in the spatial attention layer, two 1×M×M feature maps F are obtained. 4 、F 4 ';Feature map F 4 、F 4 'Spin along the channel dimension to get a feature map F of size 2×M×M 5 ; Feature map F 5 After 1×1 convolution, we get a feature map F of size 1×M×M. 6 ; Feature map F 6 After the Sigmoid activation function layer, the output of the Sigmoid activation function layer is combined with the feature map F 3 After multiplication, we get the feature map F 7 . The feature map F 7 After 3×3 convolution, we get a feature map F of size C×M / 2×M / 2 8 , feature map F 8 After the ReLU activation function layer, the output of the Sigmoid activation function layer is subjected to a 4×4 transposed convolution to obtain a feature map F of size 1×M×M. 9 ; The feature map F 9 With the input feature map F 0 The output of the TCAM module is concatenated and its size is C×M×M.

[0040] Specifically, Figure 2 As shown in Figure 1, the LTEN branch is composed of five LTE Blocks connected in series. Figure 5 As shown in the figure, each LTEBlock includes a 3×3 convolution module, an average pooling layer, a cross-space multi-scale attention module and a 1×1 convolution layer; the 3×3 convolution module includes two cascaded 3×3 convolution layers. Among them, for the second to fourth LTE Blocks, the output image scale is 1 / 2 of the input image scale, and the number of output channels is twice the number of input channels; while the output image scale of the fifth LTE Block is equal to the input image scale, and the number of output channels is twice the number of input channels.

[0041] In the present invention, the input image size of the LTEN branch is 3×H×W, where H×W is the scale, and “3” represents the three channels of the image, namely R (red), G (green), and B (blue). The output size of the first LTE Block is 16×H / 2×W / 2, the output size of the second LTE Block is 32×H / 4×W / 4, the output size of the third LTE Block is 64×H / 8×W / 8, the output size of the fourth LTE Block is 128×H / 16×W / 16, and the output size of the fifth LTE Block is 256×H / 16×W / 16.

[0042] S4. Input the first branch feature and the second branch feature into a feature fusion module to obtain a fusion feature.

[0043] Specifically, Figure 3 As shown in Figure 1, the feature fusion module GFM includes a gating network, which includes two 1×1 convolution paths and a Sigmoid activation function that generates the gating weights of each feature channel. The processing of the feature fusion module includes:

[0044] S41. Input the first branch feature into a 1×1 convolution channel to obtain a first convolution feature;

[0045] S42. Input the second branch feature into another 1×1 convolution channel to obtain a second convolution feature;

[0046] S43. The first convolution feature and the second convolution feature are concatenated and input into the Sigmoid activation function layer to obtain the first branch gating weight; the second branch gating weight is obtained by subtracting the first branch gating weight from 1;

[0047] S44. The first branch feature is multiplied by the first branch gating weight to obtain the first feature, and the second branch feature is multiplied by the second branch gating weight to obtain the second feature; the first feature and the second feature are concatenated to obtain a fusion feature.

[0048] S5. Input the terrain text corresponding to the image into the text prompt encoder to obtain text features.

[0049] Specifically, the text affine layer is a small fully connected network (FCN) consisting of a linear transformation layer (the output dimension size is 1 / 2 of the input dimension size), a Relu activation function, and a batch normalization layer.

[0050] Preferably, in an embodiment of the present invention, the size of each image in the preprocessed image data is 3×512×512, so the input of the MTG-SAM branch and the LTEN branch are both 3×512×512, and the output of the MTG-SAM branch and the LTEN branch is 256×32×32; the output size of the GFM module is 256×32×32. The CLIP text encoder receives text as input, and the output text feature scale is 1×512, and the text affine layer output scale is 1×256.

[0051] S6. Input the fused features and text features into the Mask Decoder to obtain the segmentation result; calculate the loss training model according to the segmentation result and the mask image until the model parameters converge.

[0052] In one embodiment, the present invention and other methods are quantitatively experimented on the Mars image dataset AI4Mars, and the experimental results are shown in Table 1. It can be seen that the segmentation method based on convolutional neural network performs poorly. UNet has the best recognition effect on "bigrock", but the overall performance is average. UNext has improvements on "bedrock" and "soil", but performs poorly on "bigrock". The performance of DeepLabV3 is between the two. The Transformer-based methods, MedT and TransUNet, perform poorly due to insufficient data set size and category imbalance. In addition, SAM is also not suitable for Mars image segmentation tasks in the zero-sample case (SAM-ZS); while SAM-Adapter and AdaptiveSAM using model fine-tuning technology show performance superior to traditional segmentation methods. In the present invention, the best performance on most terrain categories is achieved when only the MTG-SAM branch is used. After adding the LTEN branch, the performance is significantly improved, and the best segmentation effect is achieved.

[0053] Table 1 Experimental results

[0054]

[0055] In Table 1, the best results are shown in bold, and the suboptimal results are shown in underline.

[0056] On the AI4Mars dataset, the segmentation effects of the present invention and other SAM-based methods are compared visually. Figure 6 As shown, the left side is the input text prompt. It can be seen that the segmentation results of SAM-Adapter and AdaptiveSAM have the problems of over-prediction or under-prediction of terrain and blurred terrain boundaries. The present invention has significantly improved the above problems, and the results are closest to the real annotation.

[0057] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0058] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A text-guided multi-view Mars terrain feature image segmentation method based on SAM, characterized in that: Construct and train the MTSNet model, input the Mars image to be segmented and the terrain prompt text into the trained MTSNet model to obtain the segmentation result; the MTSNet model includes an MTG-SAM branch, an LTEN branch, a feature fusion module, a text prompt encoder and a MaskDecoder; the MTG-SAM branch is composed of a plurality of Transformer Blocks in cascade, and a TCAM module is introduced into each TransformerBlock; the text prompt encoder includes a CLIP text encoder, a text affine layer and a prompt encoder; The training process of the MTSNet model includes the following steps: S1. Obtain training image data and its corresponding terrain text prompt data for preprocessing to obtain preprocessed image data; the preprocessed data includes a plurality of image mask terrain text pairs, each image mask terrain text pair includes an image, a mask image and a terrain text; S2. Initialize the MTSNet model, freeze all parameters except the TCAM module in the MTG-SAM branch, and freeze the CLIP text encoder parameters; S3. Input the image into the MTG-SAM branch and the LTEN branch respectively to obtain the first branch feature and the second branch feature; S4. Inputting the first branch feature and the second branch feature into a feature fusion module to obtain a fusion feature; S5. Input the terrain text corresponding to the image into the text prompt encoder to obtain text features; S6. Input the fused features and text features into the Mask Decoder to obtain the segmentation result; calculate the loss training model according to the segmentation result and the mask image until the model parameters converge.

2. The text-guided multi-view Mars terrain feature image segmentation method based on SAM according to claim 1, characterized in that: Step S1 specifically includes: S11. Perform pixel standardization on each image, including: calculating the pixel mean and pixel standard deviation of the image in the three RGB channels respectively; for each pixel value of each channel in the image, subtract the pixel value from the mean of the corresponding channel and divide it by the standard deviation of the corresponding channel; S12. For each pixel-normalized image, mask processing is performed according to the corresponding terrain text prompt to obtain N image-mask terrain text pairs, where N is the number of terrain types; wherein in each mask image, the value of the pixel corresponding to the terrain to be segmented is 1, and the values ​​of the remaining pixels are 0; S13. Transform the training image data to obtain enhanced data, where the transformation operations include horizontal flipping, vertical flipping, rotation, horizontal translation of 10%, vertical translation of 10%, brightness change, and random cropping; S14. Perform the operations of steps S11-S12 on each new image in the enhanced data.

3. The text-guided multi-view Mars terrain feature image segmentation method based on SAM according to claim 1, characterized in that: The LTEN branch consists of five LTE Blocks connected in series. Each LTE Block includes a 3×3 convolution module, an average pooling layer, a cross-spatial multi-scale attention module and a 1×1 convolution layer; the 3×3 convolution module includes two cascaded 3×3 convolution layers.

4. The text-guided multi-view Mars terrain feature image segmentation method based on SAM according to claim 1, characterized in that: Each Transformer Block includes a cascaded first LN layer, a multi-head attention layer, a TCAM module, a second LN layer, and an MLP layer. There is a residual branch between the Transformer Block input and the TCAM module output, and there is a residual branch between the second LN layer input and the MLP layer output.

5. The text-guided multi-view Mars terrain feature image segmentation method based on SAM according to claim 1, characterized in that: The TCAM module includes a channel attention layer, a spatial attention layer and a spatial transformation layer; the channel attention layer includes global maximum pooling, global average pooling, MLP and Sigmoid activation function layers; the spatial attention layer includes global maximum pooling, global average pooling, channel splicing layer, 1×1 convolution layer and Sigmoid activation function layer; the spatial transformation layer includes a downsampling layer, a ReLU activation function layer and an upsampling layer, in which the downsampling layer adopts 3×3 convolution and the upsampling layer adopts 4×4 device convolution.

6. The text-guided multi-view Mars terrain feature image segmentation method based on SAM according to claim 1, characterized in that: The feature fusion module includes two 1×1 convolution channels and a Sigmoid activation function layer; The processing of the feature fusion module includes: S41. Input the first branch feature into a 1×1 convolution channel to obtain a first convolution feature; S42. Input the second branch feature into another 1×1 convolution channel to obtain a second convolution feature; S43. The first convolution feature and the second convolution feature are concatenated and input into the Sigmoid activation function layer to obtain the first branch gating weight; the second branch gating weight is obtained by subtracting the first branch gating weight from 1; S44. The first branch feature is multiplied by the first branch gating weight to obtain the first feature, and the second branch feature is multiplied by the second branch gating weight to obtain the second feature; the first feature and the second feature are concatenated to obtain a fusion feature.

7. The text-guided multi-view Mars terrain feature image segmentation method based on SAM according to claim 1, characterized in that: The text affine layer is a small fully connected network consisting of a linear transformation layer, a Relu activation function, and a batch normalization layer.

Images