Ice semantic segmentation method for remote sensing images in different modes
By introducing a channel adaptation mechanism and a Transformer encoder, the problem of unified processing of remote sensing images of different modalities is solved, achieving efficient semantic segmentation of ice formations, improving the model's adaptability and segmentation accuracy, and making it suitable for ice condition monitoring and segmentation tasks on various remote sensing platforms.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-24
- Publication Date
- 2026-03-27
AI Technical Summary
Existing deep learning models cannot process remote sensing images of different modalities simultaneously. They need to be trained and deployed separately for each modality, which increases system complexity and computational cost. Furthermore, forced data transformation can lead to the loss of key spectral information and reduce segmentation accuracy.
By introducing a channel adaptation mechanism, a semantic segmentation method for ice formations in remote sensing images of different modalities is designed. The NDWI value is used to replace the green band, and 3D and 2D convolution processing are combined to perform unified modeling and feature extraction through a Transformer encoder. This enables the extraction and alignment of spatially related features across channels, and finally, multi-scale feature fusion and decoding are performed.
It enables unified processing of multiple modal remote sensing images in the same model, improving the model's versatility and segmentation accuracy, and is suitable for air situation monitoring and segmentation tasks on various remote sensing platforms.
Smart Images
Figure CN121746701A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this application relate to the fields of artificial intelligence and deep learning technology, and in particular to a method for semantic segmentation of ice formations in remote sensing images of different modalities. Background Technology
[0002] Ice floes are a special hydrological phenomenon that forms on rivers in cold regions during winter freezing, typically manifesting as ice blocks floating or accumulating on the water surface. Their distribution and thickness are influenced by various factors such as temperature, water flow, latitude, and topography, serving as crucial data for ice condition evolution analysis and ice prevention scheduling. In recent years, with the development of deep learning technology, especially semantic segmentation, research teams have begun using semantic segmentation to perform pixel-level segmentation of remote sensing images, outputting the category of each pixel and estimating its spatial attributes such as area, boundary, and length. This provides data support for ice condition assessment and early warning. Compared to traditional spectral index-based methods, semantic segmentation offers higher accuracy and robustness.
[0003] However, existing ice segmentation models have limitations when processing remote sensing images of different modalities. In practical applications, remote sensing images often originate from different sensor platforms, such as high-resolution three-channel RGB images acquired by UAVs and four-channel multispectral images including near-infrared bands acquired by satellites. These two types of data differ significantly in the number of channels, spectral information, spatial resolution, and imaging perspective.
[0004] Current deep learning models are typically designed for specific modalities, meaning a single model can only handle inputs with a fixed number of channels. When processing data from different modalities, models need to be trained and deployed separately for each modality, and these algorithms have achieved certain results. For example, Chen et al. proposed a model called GLNet, which is trained on RGB data for semantic segmentation of remote sensing images. Hong et al. created a general-purpose remote sensing foundation model, SpectralGPT, specifically designed for processing spectral remote sensing images. However, in practical applications, the specific implementations of these models still require separate training and inference for different modalities. A single model cannot be used to process data from different modalities in a single training iteration. This not only increases system complexity and computational cost but also prevents the model from comprehensively utilizing the complementary information from different modalities. Furthermore, forcibly converting multi-channel data to three-channel data (e.g., discarding the NIR bands of multispectral images) results in the loss of crucial spectral information, thereby reducing segmentation accuracy. Therefore, there is an urgent need in this field for a semantic segmentation model that can be compatible with different modal inputs and achieve unified processing, in order to improve its versatility and practicality in different modal remote sensing scenarios. Summary of the Invention
[0005] To address the aforementioned technical issues, embodiments of this application propose an ice crystal semantic segmentation method for remote sensing images of different modalities. Without altering the core structure of the model, a channel adaptation mechanism is introduced to achieve automatic identification and processing of heterogeneous input data. This method supports simultaneous training and prediction of remote sensing images of multiple modalities within the same model, and the number of modalities can be expanded according to actual needs, thereby enhancing the model's versatility and extensibility in different modal remote sensing scenarios.
[0006] To achieve the above objectives, embodiments of this application propose a semantic segmentation method for ice formation in remote sensing images of different modalities. The method includes: identifying the number of channels and data types of the acquired remote sensing images, classifying them into three-channel RGB images and four-channel multispectral images; directly normalizing the three-channel RGB images; calculating the NDWI value for the four-channel multispectral images, replacing the original green band with the NDWI value to enhance the spectral difference between water and ice formations, and then performing normalization; inputting the normalized remote sensing images into a channel adaptation module for unified modeling; extracting cross-channel spatial correlation features along the channel dimension through 3D convolution processing; and then performing channel compression and gradation through 2D convolution processing. The image is aligned and unified into a three-channel feature map. The three-channel feature map is input into the Transformer encoder, which first divides the image into fixed-size blocks, obtains a token sequence through linear mapping, and adds position encoding. Then, the token sequence is passed through a multi-layer self-attention unit and a feedforward network to extract global feature representations. The multi-scale features output from each layer of the Transformer encoder are input into a multi-layer feature aggregation decoder for decoding and reconstruction. The multi-scale features are upsampled, aligned, and fused, and the spatial resolution is gradually restored by combining low-level detail information and high-level semantic information. Finally, a semantic prediction map with the same size as the remote sensing image is output, realizing three-class semantic segmentation of ice, water, and land.
[0007] To achieve the above objectives, embodiments of this application also propose an electronic device, including a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement the ice crystal semantic segmentation method for remote sensing images of different modalities as described above.
[0008] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the ice crystal semantic segmentation method for remote sensing images of different modalities as described above.
[0009] Optionally, the three-channel RGB image adopts an 8-bit format, and the three-channel RGB image is directly normalized, including: normalizing the pixel values of each point of the three-channel RGB image to 255 to obtain the normalized RGB image.
[0010] Optionally, the four-channel multispectral image uses a 16-bit unsigned integer format. The NDWI value is first calculated for the four-channel multispectral image, and the original green band is replaced with the NDWI value to enhance the spectral difference between the water body and the ice. Then, normalization processing is performed, including: Extract the green and near-infrared bands from the four-channel multispectral image, and calculate the NDWI value based on the green and near-infrared bands. The formula for calculating the NDWI value is as follows: ; in, Indicates the near-infrared band. Indicates the green band. Indicates the NDWI value; The calculated NDWI value is used to replace the original green band of the four-channel multispectral image to obtain the replaced multispectral image. The pixel values of each point in the replaced multispectral image are normalized to 65535 to obtain the normalized multispectral image.
[0011] Optionally, 3D convolution processing is implemented by a 3D convolutional layer in the channel adaptation module. This 3D convolution processing extracts cross-channel spatially relevant features along the channel dimension, including: The normalized remote sensing image is represented as a four-dimensional tensor. , For batch size, This represents the original number of channels. and These represent the height and width of the normalized remote sensing image, respectively. Four-dimensional tensor Expanded to a five-dimensional tensor The data is then fed into a three-dimensional convolutional layer with a kernel size of 3×3×3 and a stride of 1×1×1. The 3D convolution process of the three-dimensional convolutional layer is performed simultaneously in three dimensions: channel, height, and width, ultimately yielding cross-channel spatially correlated features.
[0012] Optionally, 2D convolution processing is implemented by a two-dimensional convolutional layer in the channel adaptation module. Channel compression and format alignment are performed through 2D convolution processing, ultimately unifying the data into a three-channel feature map, including: The cross-channel spatial correlation features output by the 3D convolutional layer are transformed and reduced in dimensionality to convert them into standard 4D tensors. , The number of channels representing spatially related characteristics across channels; Four-dimensional tensor The input is fed into a two-dimensional convolutional layer with a kernel size of 3×3 and a stride of 1; 2D convolution processing of two-dimensional convolutional layers transforms four-dimensional tensors The number of channels is uniformly compressed to 3 channels for output, resulting in a shape of Three-channel feature map.
[0013] Optionally, the three-channel feature map is input into the Transformer encoder, first divided into fixed-size image blocks, linearly mapped to obtain a token sequence, and then positional encoding is added. Subsequently, the token sequence is passed sequentially through multiple layers of self-attention units and a feedforward network to extract global feature representations, including: The input single-channel feature map is divided into several fixed-size image blocks. Each image block is flattened and then mapped to a token vector through linear projection. Let the size of the image block be... Then co-generated One token, Each token vector is further superimposed with a learnable location encoding vector to preserve spatial location information; The token sequence, consisting of all token vectors superimposed with position encoding vectors, is input into multiple Transformer encoding layers of the Transformer encoder. Each Transformer encoding layer includes a multi-head self-attention mechanism and a feedforward neural network, which supports long-distance information transfer between different regions, ultimately yielding multi-scale features from the output of each layer of the Transformer encoder.
[0014] Optionally, the multi-layer feature aggregation decoder contains multiple decoding layers, each of which consists of an upsampling unit, an alignment unit, and a lightweight convolutional unit; Intermediate features at each scale First, channel projection and upsampling are performed to restore the low-resolution features to the target size, and then they are aligned in the spatial dimension. Then, the deepest features are analyzed. Beginning, with the features of the next level The fusion is performed by adding elements one by one, and the fusion process proceeds from top to bottom, continuously combining features from shallower layers, and finally inputting them into lightweight convolutional units for feature smoothing. Finally, the fused features are input to the output head module and mapped to the number of channels of the category through 1×1 convolution, resulting in a pixel-level semantic prediction map with the same size as the remote sensing image. Each point in the semantic prediction map contains the probability distribution of the three categories, realizing three-class semantic segmentation of ice, water and land.
[0015] Optionally, the identification of the number of channels and data types of the acquired remote sensing images is implemented by the image preprocessing module. The image preprocessing module, channel adaptation module, Transformer encoder, multi-layer feature aggregation decoder and output head module together form the ice crystal semantic segmentation model.
[0016] This application proposes an ice jam semantic segmentation method for remote sensing images of different modalities. It can simultaneously adapt to three-channel RGB images and four-channel multispectral images, and supports simultaneous input and training of remote sensing images of different modalities in the same training batch, eliminating the need to build and deploy multiple models separately. This significantly improves the model's adaptability, generalization ability, and engineering efficiency. The channel adaptation module designed in this application combines NDWI calculation with green band replacement, 3D convolution, and 2D convolution operations, effectively enhancing information fusion and feature alignment capabilities between different channels, and improving the model's segmentation accuracy in complex ice jam and water body boundary regions. This application introduces a Transformer-based global modeling and MLA multi-scale feature decoding mechanism, which can simultaneously capture long-distance contextual information and detailed edge features, improving the model's perception of ice jam region boundaries. It is suitable for ice jam monitoring and segmentation tasks on various remote sensing platforms. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.
[0018] Figure 1 This is a flowchart of an ice crystal semantic segmentation method for remote sensing images of different modalities provided in one embodiment of this application; Figure 2 This is a structural diagram of ice crystal semantic segmentation provided in one embodiment of this application; Figure 3 This is a structural diagram of the Vision Transformer block provided in one embodiment of this application; Figure 4 This is a structural diagram of a Transformer encoder provided in one embodiment of this application; Figure 5 This is a structural diagram of a multi-scale feature coding module provided in one embodiment of this application; Figure 6This is a comparison image on the NWPU_YRCC1 dataset provided in one embodiment of this application; Figure 7 This is a comparison image on the GF_ICE dataset provided in one embodiment of this application; Figure 8 This is a comparison chart of the output on the NWPU_YRCC1 dataset and the original input provided in one embodiment of this application; Figure 9 This is a comparison chart of the output on the GF_ICE dataset and the original input provided in one embodiment of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.
[0020] To address the technical challenge of traditional models being unable to simultaneously adapt to three-channel RGB images and four-channel multispectral images, one embodiment of this application proposes an ice crystal semantic segmentation method for remote sensing images of different modalities, applied to an electronic device. The electronic device can be a terminal or a server; this embodiment and subsequent embodiments will use a server as an example. The implementation details of the ice crystal semantic segmentation method for remote sensing images of different modalities proposed in this embodiment are described below. These details are provided for ease of understanding and are not essential for implementing this solution. The ice crystal semantic segmentation method for remote sensing images of different modalities proposed in this embodiment can be abbreviated as MixFormer.
[0021] The specific process of the ice crystal semantic segmentation method for remote sensing images of different modalities proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 11: Identify the number of channels and data types of the acquired remote sensing images, distinguishing them into three-channel RGB images and four-channel multispectral images. Perform normalization processing directly on the three-channel RGB images. For the four-channel multispectral images, first calculate the NDWI value, replace the original green band with the NDWI value to enhance the spectral difference between the water body and the ice, and then perform normalization processing.
[0022] like Figure 2 As shown in the figure, the ice crystal semantic segmentation method proposed in this embodiment, applicable to remote sensing image inputs of different modalities, constructs a unified deep neural network structure (ice crystal semantic segmentation model), mainly including an image preprocessing module, a channel adaptation module, a Transformer encoder module, a multi-scale feature decoding module, and an output head module. By introducing a channel dynamic adaptation mechanism, remote sensing images from different sources, with different numbers of channels, and different bit depths can all be trained and inferred in the same model, improving the model's adaptability and deployment flexibility in multimodal remote sensing scenarios. This embodiment uses three-channel RGB images and four-channel multispectral images (using four-channel multispectral images acquired by Gaofen series satellites as examples) for illustration. In practical applications, it can adapt to remote sensing image inputs with any number of channels.
[0023] In the specific implementation, the image preprocessing module identifies the number of channels and data types of the acquired remote sensing images (also known as data type and bit depth discrimination), distinguishing them into three-channel RGB images and four-channel multispectral images. The three-channel RGB images (usually in 8-bit format, uint 8) are directly normalized. For the four-channel multispectral images (usually in 16-bit unsigned integer format, uint16), the NDWI value is first calculated, and the original green band is replaced with the NDWI value to enhance the spectral difference between the water body and the ice floes, and then normalization is performed.
[0024] In one example, the image preprocessing module normalizes the pixel values of each point in the three-channel RGB image to 255, resulting in a normalized RGB image.
[0025] In one example, the image preprocessing module extracts the green and near-infrared bands from a four-channel multispectral image, and calculates the NDWI value based on the green and near-infrared bands. The formula for calculating the NDWI value is as follows: ; in, Indicates the near-infrared band. Indicates the green band. This represents the NDWI value.
[0026] The calculated NDWI values are then used to replace the original green bands in the four-channel multispectral image, resulting in a replaced multispectral image. Finally, the pixel values of each point in the replaced multispectral image are normalized to 65535, resulting in a normalized multispectral image. In this embodiment, the NDWI calculation results replace the green bands in the original image, facilitating subsequent channel adaptation and unified modeling.
[0027] Step 12: Input the normalized remote sensing image into the channel adaptation module for unified modeling. Extract cross-channel spatial features along the channel dimension through 3D convolution processing, and then perform channel compression and format alignment through 2D convolution processing to finally unify it into a three-channel feature map.
[0028] In the specific implementation, the normalized remote sensing image will be input into the channel adaptation module for unified modeling. The channel adaptation module extracts cross-channel spatially related features along the channel dimension through 3D convolution processing, and then performs channel compression and format alignment through 2D convolution processing, finally unifying them into a three-channel feature map.
[0029] In one example, 3D convolution processing is implemented by a 3D convolutional layer in the channel adaptation module, which represents the normalized remote sensing image as a four-dimensional tensor. , For batch size, This represents the original number of channels. and These represent the height and width of the normalized remote sensing image, respectively. To fully exploit the joint spatial features between different channels, the channel adaptation module uses a four-dimensional tensor... Expanded to a five-dimensional tensor The data is then fed into a three-dimensional convolutional layer with a kernel size of 3×3×3 and a stride of 1×1×1.
[0030] The 3D convolution process of the three-dimensional convolutional layer is performed simultaneously in three dimensions: channel, height, and width, ultimately yielding cross-channel spatially correlated features.
[0031] In one example, the 2D convolution processing is implemented by a 2D convolutional layer in the channel adaptation module. The channel adaptation module performs dimensionality transformation and dimensionality reduction on the cross-channel spatially related features output by the 3D convolutional layer, converting them into a standard 4D tensor. , The number of channels represents the spatially related features across channels, and then the four-dimensional tensor is... Channel compression is achieved by feeding the input into a two-dimensional convolutional layer with a kernel size of 3×3 and a stride of 1.
[0032] 2D convolution processing of two-dimensional convolutional layers transforms four-dimensional tensors The number of channels is uniformly compressed to 3 channels for output, resulting in a shape of The three-channel feature map will be used as input to the Transformer encoder to achieve feature alignment and unified representation of remote sensing images of different modalities.
[0033] Step 13: Input the three-channel feature map into the Transformer encoder. First, divide it into fixed-size image blocks, obtain the token sequence through linear mapping, and add position encoding. Then, pass the token sequence through multiple layers of self-attention units and feedforward network to extract global feature representation.
[0034] In the specific implementation, the three-channel feature map is input to the Transformer encoder. The Transformer encoder first divides the three-channel feature map into fixed-size image blocks, obtains a token sequence through linear mapping, and adds positional encoding. Then, the token sequence is passed sequentially through multiple layers of self-attention units and a feedforward network to extract global feature representations. The Transformer encoder (e.g., ...) Figure 4 As shown) adopts a Vision Transformer-based block (such as... Figure 3 The structure shown can extract non-local features from images and capture long-distance semantic dependencies.
[0035] In one example, the input single-channel feature map is divided into several fixed-size image blocks. Each image block is flattened and then mapped to a token vector through linear projection. Let the size of the image block be... Then co-generated One token, Each token vector is further superimposed with a learnable location encoding vector to preserve spatial location information.
[0036] A token sequence consisting of all token vectors superimposed with positional encoding vectors is input into multiple Transformer encoding layers of the Transformer encoder. Each Transformer encoding layer includes a multi-head self-attention mechanism and a feedforward neural network, supporting long-distance information transfer between different regions. This ultimately yields multi-scale features from each layer of the Transformer encoder. The outputs of each layer of the Transformer encoder are then used in a subsequent multi-scale feature aggregation module to achieve hierarchical modeling from low-level details to high-level semantics.
[0037] Step 14: Input the multi-scale features output from each layer of the Transformer encoder into the multi-layer feature aggregation decoder for decoding and reconstruction. Upsample, align and fuse the multi-scale features, and gradually restore the spatial resolution by combining low-level detail information and high-level semantic information. Finally, output a semantic prediction map with the same size as the remote sensing image to achieve three-class semantic segmentation of ice, water and land.
[0038] In the specific implementation, the multi-scale features output by each layer of the Transformer encoder are input into the multi-layer feature aggregation decoder for decoding and reconstruction. The multi-layer feature aggregation decoder upsamples, aligns and fuses the multi-scale features, and gradually restores the spatial resolution by combining low-level detail information and high-level semantic information. Finally, it outputs a semantic prediction map with the same size as the remote sensing image, realizing three-class semantic segmentation of ice, water and land.
[0039] In one example, the structure of a multi-layer feature aggregation decoder can be as follows: Figure 5 As shown, the multi-layer feature aggregation decoder contains multiple decoding layers, each consisting of an upsampling unit, an alignment unit, and a lightweight convolutional unit.
[0040] Intermediate features at each scale First, channel projection and upsampling are performed to restore the low-resolution features to the target size, and then they are aligned in the spatial dimension. Then, the deepest features are analyzed. Beginning, with the features of the next level The fusion process is performed by adding elements one by one, proceeding from top to bottom, continuously combining features from shallower layers, and finally inputting them into lightweight convolutional units for feature smoothing.
[0041] Finally, the fused features are input to the output head module, and mapped to the number of channels of the category through 1×1 convolution to obtain a pixel-level semantic prediction map with the same size as the remote sensing image. Each point in the semantic prediction map contains the probability distribution of the three categories, realizing three-class semantic segmentation of ice, water and land.
[0042] This embodiment proposes an ice jam semantic segmentation method for remote sensing images of different modalities. It can simultaneously adapt to three-channel RGB images and four-channel multispectral images as input, and supports simultaneous input and training of remote sensing images of different modalities in the same training batch, eliminating the need to build and deploy multiple models separately. This significantly improves the model's adaptability, generalization ability, and engineering efficiency. The channel adaptation module designed in this embodiment combines NDWI calculation with green band replacement, 3D convolution, and 2D convolution operations, effectively enhancing the information fusion and feature alignment capabilities between different channels, and improving the model's segmentation accuracy in complex ice jam and water body boundary regions. This embodiment introduces a Transformer-based global modeling and MLA multi-scale feature decoding mechanism, which can simultaneously capture long-distance contextual information and detailed edge features, improving the model's perception of ice jam region boundaries. It is suitable for ice jam monitoring and segmentation tasks on various remote sensing platforms.
[0043] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.
[0044] In one embodiment, to verify the effectiveness of the proposed ice crystal semantic segmentation method (MixFormer) for remote sensing images of different modalities, we conducted a simulation experiment.
[0045] This simulation experiment used the NWPU_YRCC1 dataset and the GF_ICE dataset for model training and testing. The NWPU_YRCC1 dataset contains 814 visible light images taken by drones along the Yellow River from Ningxia to Inner Mongolia. GF_ICE is a multispectral dataset of the Yellow River basin taken by high-resolution satellite remote sensing, containing 11 multispectral images taken by the GF2 satellite along the Yellow River from Ningxia to Inner Mongolia, which we cropped into 768×768 images.
[0046] This simulation experiment compares MixFormer with existing multimodal semantic segmentation algorithms, and the comparison results are as follows: Figure 6 , Figure 7 As shown, these algorithms include convolution-based DeepLabV3+ and U-Net++ models, as well as mainstream methods based on the Transformer architecture such as SegFormer and Mask2Former. The model segmentation results can be shown as follows: Figure 8 , Figure 9As shown. All models were evaluated on the same dataset and with the same training strategy. The main performance metric used in the experiments was mIoU (Mean Intersection over Union), which is the mean of the Intersection over Union (IoU) and is used to measure the performance of the semantic segmentation model. Among convolutional and Transformer-based semantic segmentation models, MixFormer showed competitive results, achieving 89.22% mIoU on NWPU_YRCC1 and 86.62% mIoU on GF_ICE.
[0047] Another embodiment of this application provides an electronic device, such as Figure 10 As shown, it includes a processor 21 and a memory 22. The memory 22 stores instructions that the processor 21 can execute. When the processor 21 is configured to execute the instructions, the electronic device can implement an ice crystal semantic segmentation method for remote sensing images of different modalities as described in the above method embodiment.
[0048] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.
[0049] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.
[0050] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement an ice crystal semantic segmentation method for remote sensing images of different modalities as described in the above method embodiments.
[0051] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.
[0052] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.
Claims
1. A semantic segmentation method for ice formations in remote sensing images of different modalities, characterized in that, include: The acquired remote sensing images are identified by the number of channels and data types, and are divided into three-channel RGB images and four-channel multispectral images. The three-channel RGB images are directly normalized, while the four-channel multispectral images first calculate the NDWI value, replace the original green band with the NDWI value to enhance the spectral difference between water and ice, and then perform normalization. The normalized remote sensing image is input into the channel adaptation module for unified modeling. Cross-channel spatial correlation features are extracted along the channel dimension through 3D convolution processing, and then channel compression and format alignment are performed through 2D convolution processing, finally unifying it into a three-channel feature map. The three-channel feature map is input into the Transformer encoder, first divided into fixed-size image blocks, and then a token sequence is obtained through linear mapping. Position encoding is added, and then the token sequence is passed through multiple layers of self-attention units and feedforward networks to extract global feature representations. The multi-scale features output from each layer of the Transformer encoder are input into a multi-layer feature aggregation decoder for decoding and reconstruction. The multi-scale features are upsampled, aligned and fused, and spatial resolution is gradually restored by combining low-level detail information and high-level semantic information. Finally, a semantic prediction map with the same size as the remote sensing image is output, realizing three-class semantic segmentation of ice, water and land.
2. The method for semantic segmentation of ice formations in remote sensing images of different modalities as described in claim 1, characterized in that, The three-channel RGB image uses an 8-bit format. The three-channel RGB image is directly normalized, including normalizing the pixel values of each point of the three-channel RGB image to 255, to obtain the normalized RGB image.
3. The method for semantic segmentation of ice formations in remote sensing images of different modalities as described in claim 1, characterized in that, The four-channel multispectral image uses a 16-bit unsigned integer format. First, the NDWI value is calculated for the four-channel multispectral image. This NDWI value replaces the original green band to enhance the spectral difference between the water body and the ice. Then, normalization processing is performed, including: Extract the green and near-infrared bands from the four-channel multispectral image, and calculate the NDWI value based on the green and near-infrared bands. The formula for calculating the NDWI value is as follows: ； in, Indicates the near-infrared band. Indicates the green band. Indicates the NDWI value; The calculated NDWI value is used to replace the original green band of the four-channel multispectral image to obtain the replaced multispectral image. The pixel values of each point in the replaced multispectral image are normalized to 65535 to obtain the normalized multispectral image.
4. The method for semantic segmentation of ice formations in remote sensing images of different modalities as described in claim 1, characterized in that, 3D convolution processing is implemented by a 3D convolutional layer in the channel adaptation module. Through 3D convolution processing, cross-channel spatially relevant features are extracted along the channel dimension, including: The normalized remote sensing image is represented as a four-dimensional tensor. , For batch size, This represents the original number of channels. and These represent the height and width of the normalized remote sensing image, respectively. Four-dimensional tensor Expanded to a five-dimensional tensor The data is then fed into a three-dimensional convolutional layer with a kernel size of 3×3×3 and a stride of 1×1×1. The 3D convolution process of the three-dimensional convolutional layer is performed simultaneously in three dimensions: channel, height, and width, ultimately yielding cross-channel spatially correlated features.
5. The method for semantic segmentation of ice formations in remote sensing images of different modalities as described in claim 4, characterized in that, 2D convolution processing is implemented by a two-dimensional convolutional layer in the channel adaptation module. Channel compression and format alignment are performed through 2D convolution processing, ultimately unifying the data into a three-channel feature map, including: The cross-channel spatial correlation features output by the 3D convolutional layer are transformed and reduced in dimensionality to convert them into standard 4D tensors. , The number of channels representing spatially related characteristics across channels; Four-dimensional tensor The input is fed into a two-dimensional convolutional layer with a kernel size of 3×3 and a stride of 1; 2D convolution processing of two-dimensional convolutional layers transforms four-dimensional tensors The number of channels is uniformly compressed to 3 channels for output, resulting in a shape of Three-channel feature map.
6. The method for semantic segmentation of ice formations in remote sensing images of different modalities as described in claim 5, characterized in that, The three-channel feature map is input into the Transformer encoder, first divided into fixed-size image blocks, linearly mapped to obtain a token sequence, and then positional encoding is added. Subsequently, the token sequence is passed sequentially through multiple layers of self-attention units and a feedforward network to extract global feature representations, including: The input single-channel feature map is divided into several fixed-size image blocks. Each image block is flattened and then mapped to a token vector through linear projection. Let the size of the image block be... Then co-generated One token, Each token vector is further superimposed with a learnable location encoding vector to preserve spatial location information; The token sequence, consisting of all token vectors superimposed with position encoding vectors, is input into multiple Transformer encoding layers of the Transformer encoder. Each Transformer encoding layer includes a multi-head self-attention mechanism and a feedforward neural network, which supports long-distance information transfer between different regions, ultimately yielding multi-scale features from the output of each layer of the Transformer encoder.
7. The method for semantic segmentation of ice formations in remote sensing images of different modalities as described in claim 6, characterized in that, The multi-layer feature aggregation decoder contains multiple decoding layers, each consisting of an upsampling unit, an alignment unit, and a lightweight convolutional unit; Intermediate features at each scale First, channel projection and upsampling are performed to restore the low-resolution features to the target size, and then they are aligned in the spatial dimension. Then, the deepest features are analyzed. Beginning, with the features of the next level The fusion is performed by adding elements one by one, and the fusion process proceeds from top to bottom, continuously combining features from shallower layers, and finally inputting them into lightweight convolutional units for feature smoothing. Finally, the fused features are input to the output head module and mapped to the number of channels of the category through 1×1 convolution, resulting in a pixel-level semantic prediction map with the same size as the remote sensing image. Each point in the semantic prediction map contains the probability distribution of the three categories, realizing three-class semantic segmentation of ice, water and land.
8. The method for semantic segmentation of ice formations in remote sensing images of different modalities as described in claim 7, characterized in that, The identification of the number of channels and data types in the acquired remote sensing images is achieved by the image preprocessing module. The image preprocessing module, channel adaptation module, Transformer encoder, multi-layer feature aggregation decoder and output head module together form the ice crystal semantic segmentation model.
9. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement an ice crystal semantic segmentation method for remote sensing images of different modalities as described in any one of claims 1 to 8.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement the ice slab semantic segmentation method for remote sensing images of different modalities as described in any one of claims 1 to 8.