Multitask sea ice segmentation method based on deformable space-frequency state space network

Abstract

A multi-task sea ice segmentation method based on a deformable space-frequency state space network comprises the following steps: (1) constructing input data including a synthetic aperture radar channel, a brightness temperature channel and an auxiliary channel for multi-source sea ice remote sensing image data; (2) performing double-flow feature extraction on the input data by using a deformable space-frequency encoder; (3) constructing a sea ice category text prompt library, encoding the text prompt library by using the pre-trained visual language model to obtain a text embedding vector, and calculating the pixel-level similarity between the text embedding vector and the visual features; and (4) inputting the semantic enhanced fusion features into a visual state space decoder, and modeling a long-range context dependency relationship by using a multi-direction selective scanning mechanism for decoding to obtain a multi-task segmentation result. According to the multi-task sea ice segmentation method based on the deformable space-frequency state space network, accurate segmentation of multi-task sea ice can be realized, and the precision and robustness of multi-task sea ice segmentation are improved.

Description

A Multi-Task Sea Ice Segmentation Method Based on Deformable Spatial Frequency State Space Network Technical Field

[0001] This invention relates to the field of polar sea ice monitoring technology, and in particular to a multi-task sea ice segmentation method based on a deformable space-frequency state spatial network. Background Technology

[0002] Arctic sea ice plays a crucial role in global climate regulation. Its rapid decline and complex ice conditions in recent decades make accurate and efficient automated sea ice segmentation essential for understanding its dynamic evolution and polar shipping route planning. Early methods relied on traditional machine learning classifiers to extract handcrafted features from SAR images, such as Markov random field models, Bayesian classifiers, support vector machines, and random forests. However, this reliance on handcrafted features limited their generalization ability in complex scenarios.

[0003] Deep learning has become mainstream due to its automatic feature extraction capabilities. Early work focused primarily on using convolutional neural networks (CNNs) for single-task segmentation or classification, such as adjusting LeNet for ice type classification, fusing ResNet with hollow spatial pyramid pooling to identify ice-water edges, and utilizing weight labels, semi-supervised co-training, and weakly supervised learning to improve classification. It also extended deep learning to sea ice forecasting or combined multiple techniques to capture long-range spatial dependencies and improve segmentation. However, most methods primarily estimated single sea ice parameters. To comprehensively characterize sea ice, multi-task learning frameworks have been developed, such as MMSeaIce for concurrent prediction of multiple parameters, MFDA for cross-scene classification, and MFGC-Net for multi-task fine segmentation.

[0004] Therefore, current methods still have limitations: SAR pixels are controlled by radar backscattering and affected by speckle noise, and multi-task models often ignore their characteristics; some ice types have different physical properties, but appear visually similar in SAR images; CNNs have difficulty capturing long-range dependencies, and Transformers have low computational efficiency. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the existing technology by providing a multi-task sea ice segmentation method based on a deformable space-frequency state space network. This method achieves accurate multi-task sea ice segmentation by constructing multi-channel input, dual-stream feature extraction, text-guided semantic enhancement, and multi-directional selective scanning decoding, thereby improving the accuracy and robustness of multi-task sea ice segmentation.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solution. The present invention is a multi-task sea ice segmentation method based on a deformable space-frequency state space network. The method has the following steps: (1) For multi-source sea ice remote sensing image data, construct input data including synthetic aperture radar channel, brightness temperature channel and auxiliary channel; (2) Use deformable space-frequency encoder to extract dual-stream features from the input data, capture sea ice geometry through spatial sensing unit, and extract sea ice frequency domain texture features through frequency sensing unit combined with fractional Gabor filter; (3) Construct a sea ice category text prompt library, use a pre-trained visual language model to encode the text prompt library to obtain text embedding vectors, and calculate its pixel-level similarity with visual features, and obtain semantically enhanced fusion features through text-guided semantic injection module; (4) Input the semantically enhanced fusion features into the visual state space decoder, use a multi-directional selective scanning mechanism to model long-range contextual dependencies for decoding, and obtain multi-task segmentation results of sea ice concentration, sea ice development stage and floating ice size.

[0007] Furthermore, this method uses multi-source sea ice remote sensing images as input data. Each image contains nine observation channels, specifically: two synthetic aperture radar (SAR) channels to record the microwave scattering characteristics of the sea ice surface; two brightness temperature auxiliary channels to record the thermal radiation information of the sea ice; two wind field auxiliary channels to record the horizontal component of the sea surface wind speed; and three spatiotemporal auxiliary channels to record the longitude, latitude, and observation month of the observation point, respectively. The original image has a spatial size of 512×512 pixels. Each image is uniformly cropped into four 256×256 pixel sub-image blocks. Each sub-block retains all nine channel information as the basic unit for subsequent processing.

[0008] Furthermore, this method maps the multimodal 9-channel input to a standard format that facilitates subsequent processing and splits the information into two complementary processing paths. The specific steps are as follows: (1) Channel alignment: The 9 input channels are linearly combined into 8 intermediate channels using a 1×1 convolution kernel. That is, the 9-dimensional vector at each pixel position is weighted and summed to obtain a new 8-dimensional representation; (2) Dual-stream branch: The 8 intermediate channels are divided into two groups of 4 channels each: (2.1) Spatial branch: Focuses on capturing the geometric shape and edge structure of sea ice; (2.2) Frequency domain branch: Focuses on capturing the texture periodicity and directional features of sea ice.

[0009] Furthermore, this method introduces "deformable convolution" technology, which allows the sampling position to be dynamically offset according to the image content to fit the irregular boundary of sea ice. The specific processing flow is as follows: (1) Divide the four spatial channels into two groups, with two channels in each group; (2) The first group uses 1×1 convolution to extract basic features and use them as context information; (3) The second group combines the output of the first group to predict the offset of the sampling position, and then performs deformable 3×3 convolution; (4) Combine the results of the two groups to form a spatial feature representation.

[0010] The technical problem to be solved by the present invention can also be further realized by the following technical solutions. For the above-mentioned multi-task sea ice segmentation method based on deformable space-frequency state space network, the specific processing flow of the frequency sensing unit in the method is as follows: (1) Multi-scale frequency domain feature channel configuration: The four input frequency domain channels are mapped to four different preset spectrum scale processing branches respectively, and each channel serves as the feature carrier of the corresponding scale. By applying filters with different parameters to different channels, the texture features of each channel are constrained to be extracted in a specific frequency range: the first channel is configured as a fine-scale branch: by configuring filters with small-scale parameters, the features of high-frequency textures are extracted, corresponding to the delicate surface of thin ice; the second and third channels are configured as medium-scale branches: responding to medium-frequency textures, corresponding to transitional ice types; the fourth channel is configured as a coarse-scale branch: responding to low-frequency textures, corresponding to the rough surface of thick ice / multi-year ice; (2) Generate a fractional Gabor filter bank For each scale, a set of filters is generated according to the following parameters: 4 prototype parameters: control the order of the fractional transformation, with a value range of 0.2π to 0.8π, and the initial value is random; 4 directional parameters: cover the four main directions of 0°, 45°, 90°, and 135°; each filter has both real and imaginary parts, used to capture the phase information of the texture; thus, 4×4=16 filters are generated for each scale, which together constitute the filter library for that scale; define the fractional Gabor filter for: in Direction angle ,scale Gaussian envelope: Rotation coordinates are , Direction set Even coverage , scale set Progressing by binary ; It is a fractional Fourier transform kernel. In the formula, The normalization coefficient is... Fractional parameters for controlling rotation angle , used to adjust the degree of rotation of the filter in the frequency domain to adapt to the texture characteristics of sea ice with different roughness; (3) Convolution kernel modulation starts with a learnable basic convolution kernel and modulates it using a Gabor filter: let A learnable foundation shared across all categories Convolution kernel, has One input channel and Each output channel generates a texture-adaptive convolution kernel through fractional-order Gabor filter modulation: , Represents a set of texture prototypes designed for different orientations and scales. Customized specific convolution kernels, where and The final filter bank is constructed by concatenating the convolutional kernel groups corresponding to each texture prototype. (4) Multi-scale convolution and fusion: For each scale's input channel, the convolution operation is performed using the 16 modulation convolution kernels corresponding to that scale. The frequency domain features at a given point are calculated by convolving the input feature set with the filter bank: Using the generated Gabor filter bank of the corresponding scale, convolution operation is performed on the input channel specified as that scale in step (1). Through the selective response of the filter in the frequency domain, the physical texture information corresponding to that scale is activated and extracted from the linear features of the input channel. The convolution results of the four scales are concatenated along the channel dimension to obtain the intermediate features of 64 channels. The 64 channels are compressed into the corresponding number of channels set in the layer by 1×1 convolution to output features.

[0011] Furthermore, in this method, the outputs of the spatial branch and the frequency domain branch are concatenated in the channel dimension and fused into a unified feature representation through convolution operation. At the same time, residual connection is used to directly pass the original input information to the output end to ensure that information is not lost due to multi-layer processing. The method sets up a total of 4 encoding stages. After each stage, the image size is halved through pooling operation to gradually extract multi-scale features from local to global.

[0012] Furthermore, in this method, the processing flow of the text-guided semantic injection module is as follows: (1) Construct a domain-specific text library. Based on the SIGRID-3 international sea ice standard adopted by the AI4Arctic dataset, construct text prompts for two segmentation tasks: Sea ice development stage task, 6 categories): "A SAR image of Open Water", "A SAR image of New Ice", "A SAR image of Young Ice", "A SAR image of Thin First-Year Ice", "A SAR image of Thick First-Year Ice", "A SAR image of Old Ice"; Float ice size task, 7 categories: "A SAR image of Open Water", "A SAR image of Cake Ice", "A SAR image of Small Ice Floe", "A SAR image of Medium Ice Floe", "A SAR image of Big Ice Floe", "A SAR image of Vast Ice Floe", "A SAR image of Icebergs"; After splicing the two sets of prompts, a text library containing... (2) A comprehensive semantic vocabulary of unique text anchors; (3) A text encoder that generates text semantic embeddings using the RemoteCLIP model. Extract the semantic embedding vector for each text prompt. For the k-th text prompt in the vocabulary... Its text embedding Here, D represents the embedding dimension; in this paper, D=768. All text embeddings are aggregated into a fixed text anchor matrix. ,in and The text anchors corresponding to the two tasks respectively; (3) Visual feature projection: The visual features extracted by the decoder are ,in The number of channels is 256. and The feature map has a height and width of 16×16, and contains rich texture and environmental context information; it is first used with a learnable 1×1 convolution. Project visual features into the same semantic space as the text embedding. , where L2Norm represents the L2 normalization operation; (4) Calculate the pixel-text similarity. For each pixel position on the feature map, calculate the cosine similarity between its feature vector and all text semantic embeddings to generate a pixel-text score map. , ,in This indicates a splicing operation along the channel dimension; The temperature parameter is a learnable parameter used to control the smoothness of the similarity distribution; Softmax is calculated independently for the SOD and FLOE tasks respectively to ensure the normalization of semantic responses within each task, resulting in... Include The physical meanings of the channels are as follows: Physical meaning of the semantic response map: front Each channel corresponds to a SOD category, and the pixel value of each channel represents the semantic confidence that the location belongs to the corresponding age category; (The rest of the text appears to be unrelated and possibly machine-generated gibberish.) Each channel corresponds to a FLOE category, indicating that the position belongs to the semantic confidence of the corresponding ice floe size; a high response region indicates that the visual features are highly consistent with the semantic description of the category; (5) Semantic feature fusion: the semantic response map of 13 channels is concatenated with the original 256-channel visual features in the channel dimension to obtain 269-channel fused features, and then the number of channels is compressed back to 256 through 1×1 convolution to complete semantic injection: ,in For the spliced The channel features, after fusion, possess the following characteristics: semantic alignment: the features at each spatial location explicitly encode their semantic similarity to each category; discriminative enhancement: in visually confusing regions (such as the transition zone between thin and thick annual ice), the semantic response provides additional discriminative cues; multi-task collaboration: the simultaneous injection of semantic information from SOD and FLOE promotes the sharing of discriminative features between the two tasks.

[0013] Furthermore, in this method, the decoder processing flow is as follows: Each decoder block contains two parallel paths: Path 1: Local feature extraction. Input features first pass through a double convolutional layer; the double convolutional layer extracts local texture and edge information within a 3×3 neighborhood, fusing skip connection features with upsampled features; the output is a local feature representation. Path 2: Global dependency modeling feeds the output of the double convolution into the VSS layer. The VSS employs a selective state-space mechanism, and its discrete state-space equation is as follows: , , in Discretize the parameters for the hidden state. , , From input Dynamic computation enables the model to selectively remember and forget information, achieving effective modeling of long-range dependencies. VSS employs a four-directional selective scanning mechanism, allowing each pixel to perceive the global context at the current resolution. At resolutions of 32×32, 64×64, 128×128, and 256×256, VSS remodels long-range dependencies, ensuring the internal consistency of large structures. The output is globally perceptive features. Feature fusion: combining local features and global features The layers are concatenated along the channel dimension and then fused together using a 1×1 convolution to obtain the final output. ,in This indicates a channel-level concatenation operation. The fused feature map is formed by the network adaptively learning the fusion weights of local and global information.

[0014] Furthermore, in this method, the decoder processes the following stages: Stage 4: Upsample the bottleneck layer output to 32×32 and concatenate it with the features from Stage 4 of the encoder; use VSSDecoderBlock to establish segmentation coherence of large-scale structures at a coarser resolution; Stage 3: Upsample to 64×64 and concatenate it with the features from Stage 3 of the encoder; begin to recover medium-scale spatial details and maintain the internal consistency of large ice floes; Stage 2: Upsample to 128×128 and concatenate it with the features from Stage 2 of the encoder; recover most of the spatial resolution, refine ice block boundaries and internal texture differences; process the complete structure of small and medium-sized ice floes; Stage 1: Upsample to the original size of 256×256 and concatenate it with the features from Stage 1 of the encoder; recover pixel-level class boundaries while maintaining full-scale segmentation coherence through VSS; output the final 32-channel decoded features.

[0015] Compared with existing technologies, this invention constructs input data including synthetic aperture radar, brightness temperature, and auxiliary channels, integrates multi-source information to comprehensively characterize sea ice properties, employs a deformable spatial-frequency encoder for dual-stream feature extraction, captures geometric morphology and extracts frequency domain textures using spatial and frequency sensing units respectively, dynamically adjusts the receptive field using a deformable mechanism to adapt to complex sea ice morphology, constructs a sea ice category text prompt library, integrates text semantics into visual features to enhance semantic expression, and finally inputs the semantically enhanced features into the decoder, performs multi-directional scanning to model long-range dependencies, and improves the accuracy and consistency of multi-task segmentation. Attached Figure Description

[0016] Figure 1 is a diagram of the TDV-Net architecture of the present invention; Figure 2 is a diagram of the deformable space-frequency architecture of the present invention; Figure 3 is a diagram of the spatial sensing unit architecture of the present invention; Figure 4 is a diagram of the frequency sensing unit architecture of the present invention; Figure 5 is a diagram of the text-guided semantic injection architecture of the present invention; Figure 6 is a performance comparison diagram of the segmentation model of the present invention and the prior art. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Referring to Figures 1-6, a multi-task sea ice segmentation method based on a deformable space-frequency state-space network includes: 1. Preprocessing of input data: Multi-source sea ice remote sensing images are used as input data. Each image contains 9 observation channels, specifically: 2 synthetic aperture radar (SAR) channels: recording the microwave scattering characteristics of the sea ice surface; 2 brightness temperature auxiliary channels: recording the thermal radiation information of the sea ice; 2 wind field auxiliary channels: recording the horizontal component of the sea surface wind speed; 3 spatiotemporal auxiliary channels: recording the longitude, latitude, and observation month of the observation point, respectively.

[0019] The original image has a spatial size of 512×512 pixels. To improve computational efficiency, this invention first uniformly crops each image into four 256×256 pixel sub-image blocks, with each sub-block retaining all nine channel information as the basic unit for subsequent processing.

[0020] II. The DSF module's input channel alignment and dual-stream branching are used to uniformly map the multimodal 9-channel input to a standard format that facilitates subsequent processing, and to split the information into two complementary processing paths. Specific steps: Channel alignment: A 1×1 convolution kernel (i.e., single-point convolution) is used to linearly combine the 9 input channels into 8 intermediate channels. This operation is equivalent to weighted summation of the 9-dimensional vector at each pixel location, resulting in a new 8-dimensional representation.

[0021] Dual-stream branching: The eight intermediate channels are divided into two groups of four channels each: Spatial branch: focuses on capturing the geometric shape and edge structure of sea ice; Frequency domain branch: focuses on capturing the texture periodicity and directional features of sea ice.

[0022] As shown in Figure 2, the advantage of this dual-flow design is that the spatial branch is good at identifying local geometric features such as ice block boundaries, while the frequency domain branch is good at identifying regular patterns of ice surface texture. The two complement each other to obtain a more comprehensive feature expression.

[0023] III. Spatial Sensing Unit (DSPU) Purpose: To address the irregular shapes and blurred boundaries in sea ice images, this unit employs a convolution operation that can adaptively adjust the sampling position to enhance the perception of complex geometric structures.

[0024] Core idea: Traditional convolution operations use fixed grid points for sampling. This invention introduces "deformable convolution" technology, which allows the sampling position to dynamically shift according to the image content, thereby better conforming to the irregular boundaries of sea ice.

[0025] The processing flow is shown in Figure 3: The four spatial channels are divided into two groups (two channels in each group); the first group uses 1×1 convolution to extract basic features and use them as context information; the second group combines the output of the first group to predict the offset of the sampling position, and then performs deformable 3×3 convolution; the results of the two groups are merged to form a spatial feature representation.

[0026] IV. Frequency Sensing Unit (FPU) 4.1 Sea ice remote sensing images contain rich texture information, and these texture features are closely related to the physical properties of sea ice: (1) Texture periodicity and ice surface structure Sea ice surfaces often exhibit regular texture patterns, such as the periodic arrangement of ice ridges and the distribution of inter-ice channels. These periodic textures reflect the formation process and dynamic characteristics of sea ice.

[0027] (2) Surface roughness and ice type identification: In synthetic aperture radar (SAR) images, different ice types exhibit significantly different surface roughness characteristics:

[0028] The difference in surface roughness stems from the growth history of sea ice: thin ice forms in a short time and has a relatively smooth surface; while thick ice has undergone multiple freezing-melting cycles and dynamic deformation, resulting in a rougher and more uneven surface.

[0029] (3) The aim is to capture the directionality, periodicity and roughness characteristics of sea ice texture simultaneously through frequency domain analysis, so as to provide a discriminative feature representation for subsequent sea ice development stage (SOD) classification.

[0030] 4.2 Core Idea This invention uses a fractional-order Gabor filter as the core tool for frequency domain feature extraction.

[0031] Advantages of Gabor filters: Gabor filters are a classic texture analysis tool that exhibits highly selective responses to texture patterns in specific directions and frequencies. Specifically: Directional selectivity: It can detect linear structures such as ice ridges and crevasses with specific orientations; Frequency selectivity: It can distinguish the coarseness of textures, with low-frequency responses corresponding to coarse textures (thick ice) and high-frequency responses corresponding to fine textures (thin ice); Scale adaptability: By adjusting the scale parameters, it can cover multi-level texture features from local details to overall structure.

[0032] Innovation in Fractional Fourier Transform Extension: Traditional Gabor filters use a fixed integer transform order. This invention introduces the concept of fractional Fourier transform, extending the transform order to a learnable continuous parameter (prototype parameter α). The advantages of this extension are: enhanced filter expressiveness for intermediate frequency components; a richer filter library can be constructed through combinations of multiple prototype parameters; and the prototype parameters can be adaptively optimized during training to adapt to the specific distribution of sea ice textures.

[0033] 4.3 Processing flow. The specific processing flow of the frequency sensing unit is shown in Figure 4: Step 1: Multi-scale frequency domain feature channel configuration. The four input frequency domain channels are mapped to four different preset spectrum scale processing branches, and each channel serves as the feature carrier of the corresponding scale. By applying filters with different parameters to different channels, each channel is constrained to extract texture features within a specific frequency range: the first channel is configured as a fine-scale branch: by configuring filters with small-scale parameters, features responding to high-frequency textures are extracted, corresponding to the delicate surface of thin ice; the second and third channels are configured as medium-scale branches: responding to mid-frequency textures, corresponding to transitional ice types; the fourth channel is configured as a coarse-scale branch: responding to low-frequency textures, corresponding to the rough surface of thick ice / multi-year ice; Step 2: Generate a fractional-order Gabor filter bank. For each scale, a set of filters is generated according to the following parameters: 4 prototype parameters (α): controlling the order of the fractional-order transform, with values ​​ranging from 0.2π to 0.8π, and initial values ​​are random; 4 direction parameters (θ): covering the four main directions of 0°, 45°, 90°, and 135°; each filter has both real and imaginary parts, which can capture the phase information of the texture.

[0034] Thus, 4×4=16 filters are generated at each scale, which together constitute the filter library for that scale.

[0035] Define a fractional-order Gabor filter For: Among them Direction angle ,scale Gaussian envelope: Rotation coordinates are , Direction set Even coverage , scale set Progressing by binary It is a fractional Fourier transform kernel. In the formula, The normalization coefficient is... Fractional parameters for controlling rotation angle This is used to adjust the degree of rotation of the filter in the frequency domain to adapt to the texture characteristics of sea ice with different roughness; Step 3: Convolution kernel modulation, starting from a learnable basic convolution kernel, modulates it using a Gabor filter: Let A learnable foundation shared across all categories Convolution kernel, has One input channel and Each output channel generates a texture-adaptive convolution kernel through fractional-order Gabor filter modulation: , Represents a set of texture prototypes designed for different orientations and scales. Customized specific convolution kernels, where and This modulation mechanism introduces a structural bias into the convolution kernel, enabling the shared basic kernel energy to dynamically evolve into a specialized feature extractor adapted to the roughness and scattering frequency of specific sea ice types. The final filter bank is constructed by concatenating the convolution kernel groups corresponding to each texture prototype. Step 4: Multi-scale Convolution and Fusion. For each scale's input channel, perform convolution operations using the 16 modulation convolution kernels corresponding to that scale. The frequency domain features at a given point are calculated by convolving the input feature set with the filter bank: Using the generated Gabor filter bank of the corresponding scale, convolution operation is performed on the input channel specified as that scale in step (1). Through the selective response of the filter in the frequency domain, the physical texture information corresponding to that scale is activated and extracted from the linear features of the input channel. The convolution results of the four scales are concatenated along the channel dimension to obtain the intermediate features of 64 channels. The 64 channels are compressed into the corresponding number of channels set in the layer by 1×1 convolution to output features.

[0036] 4.4 Correlation with Ice Seed Identification: Features extracted by the spectral sensing unit are directly related to the classification of sea ice development stages (SOD task):

[0037] By extracting frequency domain features at multiple scales and in multiple directions, the FPU can effectively encode roughness information related to ice species in SAR images, providing physically interpretable feature support for subsequent classification decisions.

[0038] V. Dual-stream fusion and multi-scale coding fusion strategy: The outputs of the spatial and frequency domain branches are concatenated along the channel dimension and fused into a unified feature representation through convolution operations. Simultaneously, residual connections are used to directly pass the original input information to the output, ensuring that information is not lost due to multi-layer processing.

[0039] Multi-scale encoding: The above processing constitutes one stage of the encoder. This invention sets up four encoding stages. After each stage, the image size is halved through a pooling operation, progressively extracting multi-scale features from local to global. The channel configuration of each stage is shown in the table below:

[0040] VI. The semantic fusion layer, after four encoding stages, gradually compresses the feature map size from 256×256 to 16×16, and expands the number of channels to 128. Located at the bottom of the network, the semantic fusion layer is responsible for further processing and semantic enhancement of these deep features.

[0041] 6.1 Text-Guided Semantic Injection Module (TGSI) While the DSF module has effectively extracted spatial-frequency domain features, challenges remain in identifying fine-grained sea ice categories. Certain sea ice types exhibit highly similar visual features in SAR images. For example: new ice and young ice both display relatively dark gray values ​​with weak backscattering; thin and thick one-year-old ice show a continuous transition in scattering intensity with subtle texture differences; and ice floes of different sizes are difficult to distinguish in SAR images based solely on local texture.

[0042] Relying solely on visual features for discrimination requires the network to implicitly learn class boundaries from a large number of training samples, resulting in low learning efficiency and a high risk of confusion. To address this issue, this invention introduces a Text-Guided Semantic Injection (TGSI) module, which utilizes high-level semantic knowledge from a pre-trained remote sensing vision-language model to provide explicit class orientation for visual features.

[0043] As shown in Figure 5, the basic idea of ​​the TGSI module is to convert the text description of each sea ice category into a semantic embedding vector, calculate the similarity between visual features and the semantics of each category, generate a pixel-level semantic response map, and fuse it with the original visual features. In this way, the network no longer relies solely on fuzzy backscattered signals for classification, but obtains semantic prior information such as "whether this location is more like new ice or old ice".

[0044] Processing Step 1: Construct a domain-specific text library. Based on the SIGRID-3 international sea ice standard adopted by the AI4Arctic dataset, construct text prompts for the two segmentation tasks: Sea Ice Development Stage (SOD) Task (6 categories): "A SAR image of Open Water" "A SAR image of New Ice" "A SAR image of Young Ice" "A SAR image of Thin First-Year Ice" "A SAR image of Thick First-Year Ice" "A SAR image of Old Ice" Float Ice Size (FLOE) Task (7 categories): "A SAR image of Open Water" "A SAR image of Cake Ice" "A SAR image of Small Ice Floe" "A SAR image of Medium Ice Floe" "A SAR image of Big Ice Floe" "A SAR image of Vast Ice" The template "Floe" (giant floating ice) and "A SAR image of Icebergs" (iceberg) uses "A SAR image of {category name}" instead of a single category name. This aims to activate the pre-trained model's knowledge of SAR remote sensing characteristics and improve the targeting of semantic embedding. The two sets of prompts are concatenated to form a template containing... A comprehensive semantic vocabulary of unique text anchors.

[0045] Step 2: Generate text semantic embeddings using a text encoder based on the RemoteCLIP model. Extract the semantic embedding vector for each text cue. RemoteCLIP is a vision-language model pre-trained on large-scale remote sensing image-text pairing data, possessing stronger semantic understanding capabilities in the remote sensing domain. For the k-th text cue in the vocabulary... Its text embedding Here, D represents the embedding dimension; in this paper, D=768. All text embeddings are aggregated into a fixed text anchor matrix. ,in and These correspond to the text anchors of the two tasks respectively.

[0046] Step 3: Visual Feature Projection The visual features extracted by the decoder are... ,in The number of channels is 256. and The feature map has a height and width of 16×16, containing rich texture and environmental context information, but lacking explicit semantic labeling. To inject semantic prior, a learnable 1×1 convolution is first used. Project visual features into the same semantic space as the text embedding. , where L2Norm represents the L2 normalization operation.

[0047] Step 4: Calculate pixel-text similarity. For each pixel location on the feature map, calculate the cosine similarity between its feature vector and all text semantic embeddings to generate a pixel-text score map. , ,in This indicates a splicing operation along the channel dimension; The temperature parameter is a learnable parameter used to control the smoothness of the similarity distribution; Softmax is calculated independently for the SOD and FLOE tasks respectively to ensure the normalization of semantic responses within each task. The generated... Include The physical meanings of the channels are as follows: Physical meaning of the semantic response map: front Each channel corresponds to a SOD category, and the pixel value of each channel represents the semantic confidence that the location belongs to the corresponding age category; (The rest of the text appears to be unrelated and possibly machine-generated gibberish.) Each channel corresponds to a FLOE category, indicating the semantic confidence that the location belongs to the corresponding ice floe size; high response regions indicate that the visual features are highly consistent with the semantic description of the category.

[0048] Step 5: Semantic Feature Fusion. The 13-channel semantic response map is concatenated with the original 256-channel visual features along the channel dimension to obtain a fused feature of 269 channels. Then, a 1×1 convolution is used to compress the number of channels back to 256, completing the semantic injection. ,in For the spliced The channel features, after fusion, possess the following characteristics: semantic alignment: the features at each spatial location explicitly encode their semantic similarity to each category; discriminative enhancement: in visually confusing regions (such as the transition zone between thin and thick annual ice), the semantic response provides additional discriminative cues; multi-task collaboration: the simultaneous injection of semantic information from SOD and FLOE promotes the sharing of discriminative features between the two tasks.

[0049] This semantically enhanced representation enables the VSS module in the decoder to model long-range spatial dependencies based on category-aligned features rather than the obscured original backscattered signal, thereby enhancing segmentation coherence in visually confusing regions exhibiting similar radar features across different ice types.

[0050] 6.2 After semantic injection by the TGSI module, the output features of the semantic fusion layer (256 channels, 16×16) integrate the deep visual features extracted by the encoder with high-level semantic priors, specifically including: local texture information (frequency domain features from the encoder FPU); local geometric structure (spatial features from the encoder DSU); and semantic orientation (category priors from TGSI).

[0051] These deeply fused semantic features will be fed into the decoder for subsequent multi-scale upsampling and fine segmentation.

[0052] VII. Visual State Space Decoder (VSS Decoder) 7.1 Overall Decoder Architecture The decoder is responsible for progressively upsampling the low-resolution features (16×16) of the bottleneck layer to the original input size (256×256), and fusing the skip connection features from each stage of the encoder to recover spatial details. This invention adopts a symmetrical four-stage decoding architecture, corresponding one-to-one with the four downsampling stages of the encoder:

[0053] Each decoding stage first enlarges the feature map size by a factor of 2 through transposed convolution, then concatenates it with the output of the corresponding encoder stage in the channel dimension, and finally processes it through the decoder block.

[0054] 7.2 Decoder Block Design Traditional decoders typically use only convolutional layers to extract local features, but this has limitations when dealing with large sea ice structures: the upsampling process dilutes the feature density and weakens the dependencies between distant pixels; skip connections result in local receptive field features at each stage of the encoder, which have scale differences from the global features of the bottleneck layer; and structures such as large ice floes and extended waterways may exhibit discontinuous segmentation at different resolutions.

[0055] To address this, the present invention introduces a VSS module at each stage of the decoder to re-establish global dependencies at multiple resolution levels, ensuring the segmentation consistency of large-scale structures.

[0056] The processing flow for each decoder block contains two parallel paths: Path 1: Local Feature Extraction. Input features are first processed through a double convolutional layer (two 3×3 convolutions, combined with batch normalization and ReLU activation); the double convolutional layer extracts local texture and edge information within the 3×3 neighborhood, fusing skip connection features and upsampled features; the output is a local feature representation. .

[0057] Path 2: Global dependency modeling feeds the output of the double convolution into the VSS layer. The VSS employs a selective state-space mechanism, and its discrete state-space equation is as follows: , , in Discretize the parameters for the hidden state. , , From input Dynamic computation enables models to selectively remember and forget information, achieving effective modeling of long-distance dependencies.

[0058] VSS employs a four-directional selective scanning mechanism, enabling each pixel to perceive the global context at the current resolution. At resolutions of 32×32, 64×64, 128×128, and 256×256, VSS remodels long-range dependencies to ensure the internal consistency of large structures; the output is globally perceptive features. .

[0059] Feature fusion: combining local features and global features The layers are concatenated along the channel dimension and then fused together using a 1×1 convolution to obtain the final output. ,in This indicates a channel-level concatenation operation. The fused feature map is formed by the network adaptively learning the fusion weights of local and global information.

[0060] To adapt to the spatial structure of two-dimensional images, VSS's four-directional scanning mechanism unfolds the feature map into a sequence in four directions: Direction 1: row by row from left to right (capturing horizontal ice ridges and waterways); Direction 2: column by column from top to bottom (capturing vertical boundaries and cracks); Direction 3: row by row from right to left (enhancing the perception of the right edge); Direction 4: column by column from bottom to top (enhancing the perception of the bottom region).

[0061] Each direction is scanned independently and selectively. Finally, the outputs from the four directions are summed and fused. ,in, This represents the scanning operation in the i-th direction. This multi-directional strategy ensures that each location receives information from all directions, avoiding the bias of a single scanning direction, and is more suitable for capturing the anisotropic distribution of sea ice.

[0062] 7.3 Decoder Processing in Each Stage The fourth decoding stage (32×32) upsamples the bottleneck layer output to 32×32 and concatenates it with the features from the fourth stage of the encoder; through VSSDecoderBlock processing, the segmentation coherence of large-scale structures is established at a coarser resolution; this stage mainly processes global structures such as giant ice floes and large-scale inter-ice channels.

[0063] The third decoding stage (64×64) is upsampled to 64×64 and stitched with the features from the third stage of the encoder; it begins to recover spatial details at a medium scale and maintain the internal consistency of large ice floes; VSS ensures the classification coordination between the boundary region and the interior region.

[0064] The second decoding stage (128×128) upsamples to 128×128 and splices it with the features from the second stage of the encoder; it restores most of the spatial resolution, refines the differences in ice block boundaries and internal textures, and processes the complete structure of small and medium-sized floating ice.

[0065] The first decoding stage (256×256) upsamples to the original size of 256×256 and concatenates it with the features from the first stage of the encoder; it restores the pixel-level class boundaries while maintaining the segmentation coherence at the full-scale scale through VSS; and outputs the final 32-channel decoded features.

[0066] 7.4 The 32-channel output features of the multi-task output head decoder are fed into three independent 1×1 convolutional layers to generate pixel-level class predictions for three segmentation tasks:

[0067] Each task header outputs pixel-level category logits, which are subsequently converted into a probability distribution using softmax for loss calculation and final prediction.

[0068] 7.5 Advantages of VSS Decoders Compared to traditional pure convolutional decoders, introducing a VSS decoder offers the following advantages:

[0069] This concludes the complete architecture description of TDV-Net. The network extracts multi-scale spatial-frequency features through a DSF encoder, injects semantic priors through TGSI at the bottleneck layer, and finally restores spatial details and maintains segmentation coherence at multiple resolution levels through a VSS decoder, ultimately achieving accurate pixel-level segmentation for three tasks: sea ice concentration, sea ice development stage, and sea ice size.

[0070] This invention is validated using the AI4Arctic sea ice challenge dataset, which contains 532 scenarios (512 training samples and 20 test samples), collected from January 2018 to December 2021, covering key Arctic waters in Greenland and Canada. The dataset employs a multi-source, multi-scale architecture, with all data components registered to the Sentinel-1 geometric coordinate system. Specifically, it includes: Sentinel-1 EW SAR data, AMSR2 data, ERA5 meteorological data, and sea ice maps (ground truth labels) drawn by DMI and CIS professional analysts.

[0071] To verify the performance of TDV-Net, this invention is compared with a variety of existing methods, including: classic CNN networks U-Net, SegNet, ResNet, DenseNet, PSPNet, DeepLabV3, and DeepLabV3+; Transformer architectures SegFormer

[40] , Swin Transformer, and PoolFormer; and sea ice segmentation-specific methods MFDA, U-Net-8L, MMSeaIce, and MFGC-Net. All models are evaluated on the AI4Arctic test set. Figure 6 shows the qualitative comparison results in the SIC(a), SOD(b), and FLOE(c) tasks, comparing the proposed method with the top-ranked representative CNN and Transformer architectures and SOTA methods. Due to the limited receptive field, U-Net exhibits fragmented prediction results; while DeepLabV3+, despite multi-scale feature extraction, still has the problem of class classification errors. DenseNet and PoolFormer have improved in contour depiction, but exhibit semantic ambiguity between adjacent sea ice categories, as shown by the dashed area in the figure. U-Net-8L showed a slight performance improvement, but remained susceptible to artifacts caused by speckle noise. MMSeaIce and MFGC-Net, as state-of-the-art sea ice segmentation methods, demonstrated competitive performance (overall scores of 81.70% and 81.47%, respectively), but performed poorly in maintaining the geometric integrity of large ice floes.

[0072] In comparison, TDV-Net achieved the highest overall score (84.87%) and demonstrated consistently superior results across all tasks: spatially coherent density maps in the SIC task, accurate inter-class boundaries in the SOD task, and accurate ice floe morphology in the FLOE task. This advantage stems from effective SAR-specific speckle suppression, enhanced texture discrimination through spatial-scattering fusion, and a global attention mechanism for long-range context modeling.

Claims

1. A multi-task sea ice segmentation method based on a deformable space-frequency state-space network, characterized in that: The method steps are as follows: (1) For multi-source sea ice remote sensing image data, construct input data including synthetic aperture radar channel, brightness temperature channel and auxiliary channel; (2) Use deformable space frequency encoder to extract dual-stream features from the input data, capture sea ice geometry through spatial sensing unit, and extract sea ice frequency domain texture features through frequency sensing unit combined with fractional Gabor filter. (3) Construct a text prompt library for sea ice categories, use a pre-trained visual language model to encode the text prompt library to obtain text embedding vectors, and calculate the pixel-level similarity between the text prompt library and the visual features. Obtain semantically enhanced fusion features through the text-guided semantic injection module. (4) Input the semantically enhanced fusion features into the visual state space decoder, use a multi-directional selective scanning mechanism to model long-range contextual dependencies for decoding, and obtain multi-task segmentation results for sea ice concentration, sea ice development stage and floating ice size.

2. The multi-task sea ice segmentation method based on a deformable space-frequency state-space network according to claim 1, characterized in that: This method uses multi-source sea ice remote sensing images as input data. Each image contains nine observation channels: two synthetic aperture radar (SAR) channels to record the microwave scattering characteristics of the sea ice surface; two brightness temperature auxiliary channels to record the thermal radiation information of the sea ice; two wind field auxiliary channels to record the horizontal component of the sea surface wind speed; and three spatiotemporal auxiliary channels to record the longitude, latitude, and observation month of the observation point, respectively. The original image has a spatial size of 512×512 pixels. Each image is uniformly cropped into four 256×256 pixel sub-image blocks. Each sub-block retains all nine channel information as the basic unit for subsequent processing.

3. The multi-task sea ice segmentation method based on a deformable space-frequency state-space network according to claim 2, characterized in that: This method maps the multimodal 9-channel input to a standard format that is convenient for subsequent processing and splits the information into two complementary processing paths. The specific steps are: (1) Channel alignment: The 9 input channels are linearly combined into 8 intermediate channels using a 1×1 convolution kernel, that is, the 9-dimensional vector at each pixel position is weighted and summed to obtain a new 8-dimensional representation; (2) Dual-stream branch: The 8 intermediate channels are divided into two groups of 4 channels each: (2.1) Spatial branch: Focuses on capturing the geometric shape and edge structure of sea ice; (2.2) Frequency domain branch: Focuses on capturing the texture periodicity and directional features of sea ice.

4. The multi-task sea ice segmentation method based on a deformable space-frequency state-space network according to claim 1, characterized in that: This method introduces "deformable convolution" technology, which allows the sampling position to be dynamically shifted according to the image content to fit the irregular boundary of sea ice. The specific processing flow is as follows: (1) Divide the four spatial channels into two groups, each with two channels; (2) The first group uses 1×1 convolution to extract basic features and use them as context information; (3) The second group combines the output of the first group to predict the offset of the sampling position, and then performs deformable 3×3 convolution; (4) Combine the results of the two groups to form a spatial feature representation.

5. The multi-task sea ice segmentation method based on a deformable space-frequency state-space network according to claim 1, characterized in that: In this method, the specific processing flow of the frequency sensing unit is as follows: (1) Multi-scale frequency domain feature channel configuration: The four input frequency domain channels are mapped to four different preset spectral scale processing branches, and each channel serves as a feature carrier of the corresponding scale. By applying filters with different parameters to different channels, the texture features of each channel are constrained to extract specific frequency ranges. The first channel is configured as a fine-scale branch: by configuring filters with small-scale parameters, features that respond to high-frequency textures are extracted, corresponding to the delicate surface of thin ice. The second and third channels are configured as medium-scale branches: responding to mid-frequency textures, corresponding to transitional ice types; the fourth channel is configured as a coarse-scale branch: responding to low-frequency textures, corresponding to the rough surface of thick ice / multi-year ice; (2) Generate a fractional Gabor filter bank. For each scale, a set of filters is generated according to the following parameters: 4 prototype parameters: controlling the order of the fractional transformation, with a value range of 0.2π to 0.8π, and the initial value is random; 4 directional parameters: covering the four main directions of 0°, 45°, 90°, and 135°; each filter has both real and imaginary parts, used to capture the phase information of the texture; thus, 4×4=16 filters are generated for each scale, which together constitute the filter library for that scale; Define a fractional-order Gabor filter for: in Direction angle ,scale Gaussian envelope: Rotation coordinates are , Direction set Even coverage , scale set Progressing by binary ； It is a fractional Fourier transform kernel. In the formula, The normalization coefficient is... Fractional parameters for controlling rotation angle , used to adjust the degree of rotation of the filter in the frequency domain to adapt to the texture characteristics of sea ice with different roughness; (3) Convolution kernel modulation starts with a learnable basic convolution kernel and modulates it using a Gabor filter: let A learnable foundation shared across all categories Convolution kernel, has One input channel and Each output channel generates a texture-adaptive convolution kernel through fractional-order Gabor filter modulation: ， Represents a set of texture prototypes designed for different orientations and scales. Customized specific convolution kernels, where and The final filter bank is constructed by concatenating the convolutional kernel groups corresponding to each texture prototype. (4) Multi-scale convolution and fusion: For each scale's input channel, the convolution operation is performed using the 16 modulation convolution kernels corresponding to that scale. The frequency domain features at a given point are calculated by convolving the input feature set with the filter bank: Using the generated Gabor filter bank of the corresponding scale, convolution operation is performed on the input channel specified as that scale in step (1). Through the selective response of the filter in the frequency domain, the physical texture information corresponding to that scale is activated and extracted from the linear features of the input channel. The convolution results of the four scales are concatenated along the channel dimension to obtain the intermediate features of 64 channels. The 64 channels are compressed into the corresponding number of channels set in the layer by 1×1 convolution to output features.

6. The multi-task sea ice segmentation method based on a deformable space-frequency state-space network according to claim 1, characterized in that: In this method, the outputs of the spatial branch and the frequency domain branch are concatenated in the channel dimension and fused into a unified feature representation through convolution operation. At the same time, residual connection is used to directly pass the original input information to the output end to ensure that the information is not lost due to multi-layer processing. The method sets up a total of 4 encoding stages. After each stage, the image size is halved through pooling operation to gradually extract multi-scale features from local to global.

7. The multi-task sea ice segmentation method based on a deformable space-frequency state-space network according to claim 1, characterized in that: In this method, the processing flow of the text-guided semantic injection module is as follows: (1) Construct a domain-specific text library. Based on the SIGRID-3 international sea ice standard adopted by the AI4Arctic dataset, construct text prompts for the two segmentation tasks: sea ice development stage task, 6 categories): "A SAR image of Open Water", "A SAR image of New Ice", "A SAR image of Young Ice", "A SAR image of Thin First-Year Ice", "A SAR image of Thick First-Year Ice", "A SAR image of Old Ice"; The task related to ice floe size has seven categories: "A SAR image of Open Water," "A SAR image of Cake Ice," "A SAR image of Small Ice Floe," "A SAR image of Medium Ice Floe," "A SAR image of Big Ice Floe," "A SAR image of Vast Ice Floe," and "A SAR image of Icebergs." Two sets of prompts are stitched together to form a complete image. (2) A comprehensive semantic vocabulary of unique text anchors; (3) A text encoder that generates text semantic embeddings using the RemoteCLIP model. Extract the semantic embedding vector for each text prompt. For the k-th text prompt in the vocabulary... Its text embedding Here, D represents the embedding dimension; in this paper, D=768. All text embeddings are aggregated into a fixed text anchor matrix. ,in and The text anchors corresponding to the two tasks respectively; (3) Visual feature projection: The visual features extracted by the decoder are ,in The number of channels is 256. and The feature map has a height and width of 16×16, and contains rich texture and environmental context information; it is first used with a learnable 1×1 convolution. Project visual features into the same semantic space as the text embedding. , where L2Norm represents the L2 normalization operation; (4) Calculate the pixel-text similarity. For each pixel position on the feature map, calculate the cosine similarity between its feature vector and all text semantic embeddings to generate a pixel-text score map. ， ,in This indicates a splicing operation along the channel dimension; The temperature parameter is a learnable parameter used to control the smoothness of the similarity distribution; Softmax is calculated independently for the SOD and FLOE tasks respectively to ensure the normalization of semantic responses within each task, resulting in... Include The physical meanings of the channels are as follows: Physical meaning of the semantic response map: front Each channel corresponds to a SOD category, and the pixel value of each channel represents the semantic confidence that the location belongs to the corresponding age category; (The rest of the text appears to be unrelated and possibly machine-generated gibberish.) Each channel corresponds to a FLOE category, indicating that the position belongs to the semantic confidence of the corresponding ice floe size; a high response region indicates that the visual features are highly consistent with the semantic description of the category; (5) Semantic feature fusion: the semantic response map of 13 channels is concatenated with the original 256-channel visual features in the channel dimension to obtain 269-channel fused features, and then the number of channels is compressed back to 256 through 1×1 convolution to complete semantic injection: ,in For the spliced The channel features, after fusion, possess the following characteristics: semantic alignment: the features at each spatial location explicitly encode their semantic similarity to each category; discriminative enhancement: in visually confusing regions (such as the transition zone between thin and thick annual ice), the semantic response provides additional discriminative cues; multi-task collaboration: the simultaneous injection of semantic information from SOD and FLOE promotes the sharing of discriminative features between the two tasks.

8. The multi-task sea ice segmentation method based on a deformable space-frequency state-space network according to claim 1, characterized in that: In this method, the decoder processing flow is as follows: Each decoder block contains two parallel paths: Path 1: Local feature extraction. Input features first pass through a double convolutional layer; the double convolutional layer extracts local texture and edge information within a 3×3 neighborhood, fusing skip connection features and upsampled features; the output is a local feature representation. Path 2: Global dependency modeling feeds the output of the double convolution into the VSS layer. The VSS employs a selective state-space mechanism, and its discrete state-space equation is as follows: ， , in Discretize the parameters for the hidden state. 、 、 From input Dynamic computation enables the model to selectively remember and forget information, achieving effective modeling of long-distance dependencies; VSS, through a four-directional selective scanning mechanism, allows each pixel location to perceive the global context at the current resolution; At various resolutions of 32×32, 64×64, 128×128, and 256×256, VSS remodels long-range dependencies to ensure the internal consistency of large structures. The output is a globally perceived feature. ； Feature fusion: combining local features and global features splicing along the channel dimension; The final output is obtained by fusing the components using a 1×1 convolution. ,in This indicates a channel-level concatenation operation. The fused feature map is formed by the network adaptively learning the fusion weights of local and global information.

9. The multi-task sea ice segmentation method based on a deformable space-frequency state-space network according to claim 1, characterized in that: In this method, the decoder processes each stage as follows: Stage 4 of decoding: Upsample the bottleneck layer output to 32×32 and concatenate it with the features from Stage 4 of the encoder; The VSSDecoderBlock process establishes the segmentation coherence of large-scale structures at a coarser resolution; the third decoding stage involves upsampling to 64×64 and concatenating it with the features from the third stage of the encoder. The process begins to restore spatial details at medium scales, maintaining the internal consistency of large ice floes; the second decoding stage involves upsampling to 128×128 and stitching it with the second-stage features from the encoder. Restores most of the spatial resolution, refines ice block boundaries and internal texture differences; handles the complete structure of small and medium-sized floating ice; first decoding stage: upsamples to the original size of 256×256 and stitches it with the encoder's first stage features; Restore pixel-level class boundaries while maintaining full-image-scale segmentation coherence through VSS; output 32-channel final decoded features.

Images