Remote sensing image semantic segmentation method based on fractal curve
By integrating bilinear interpolation and fractal curves with a multi-scale feature fusion strategy, the UNet3+ model addresses the challenges of cluttered backgrounds and fog in remote sensing images, achieving enhanced edge feature capture and improved segmentation accuracy for complex scenes.
Patent Information
- Application Number
- CN202411395744.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2024-12-31
AI Technical Summary
Traditional deep learning models for remote sensing image semantic segmentation struggle with cluttered backgrounds, light changes, and fog, leading to inaccurate edge feature capture and ignoring spatial relationships between pixels, limiting their ability to perform refined operations in complex scenes.
Introduce a bilinear interpolation layer in the encoder stage of the UNet3+ model to retain spatial information, use a fractal curve module for pixel reordering, and implement a multi-scale feature fusion strategy in the decoder stage to enhance feature utilization and integration, while optimizing the network structure and parameters.
The improved UNet3+ model effectively captures edge features and enhances segmentation accuracy, particularly in complex scenes, with improved pixel clustering and inter-class separability, making it suitable for processing subtle boundaries and large-area objects.
Smart Images
Figure CN119229125A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of remote sensing image semantic segmentation, and in particular to a remote sensing image semantic segmentation method based on fractal curves. Background Art
[0002] In the field of remote sensing image semantic segmentation, existing technologies are mainly based on deep learning models, such as Unet, Unet++, Unet3+, and DeeplabV3, etc. These models perform well in extracting image features and realizing object classification, but they still have some limitations when processing remote sensing images.
[0003] Although traditional deep learning models have made significant progress in image feature extraction and object classification, they still face many challenges in remote sensing image processing. These images are often affected by factors such as cluttered backgrounds, light changes, and fog, which makes it difficult for the model to accurately capture edge features, thus affecting the accuracy of segmentation. In addition, traditional models may ignore the spatial relationship between pixels when extracting features, limiting their ability to perform refined operations in complex scenes. Summary of the invention
[0004] The purpose of this application is to provide a remote sensing image semantic segmentation method based on fractal curves to solve the problem that the remote sensing images mentioned in the above background technology are usually affected by factors such as cluttered background, light changes and fog, which makes it difficult for the model to accurately capture edge features, thereby affecting the accuracy of segmentation, and the spatial relationship between pixels may be ignored during feature extraction, limiting its ability to perform refined operations in complex scenes.
[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions: a remote sensing image semantic segmentation method based on fractal curves, comprising the following steps: in the encoding stage of the UNet3+ model, the model uses a bilinear interpolation layer to replace the traditional pooling operation. This interpolation method can retain spatial information during the downsampling process and reduce the loss of details;
[0006] In the decoding stage of the UNet3+ model, a multi-scale feature fusion strategy is used to effectively integrate features at different levels through the pooling layer. The design of this pooling layer takes into account the importance differences of features at different levels and improves the efficiency of feature utilization by selectively fusing feature information, thereby obtaining an improved UNet3+ model.
[0007] Acquire remote sensing image data and construct a remote sensing image dataset, divide it into training set and validation set in proportion, and use the improved UNet3+ model to predict and classify the remote sensing image data;
[0008] The training samples are used as input data of the improved UNet3+ network to train the improved UNet3+ model. During the training process, the model with the best effect on the validation set is taken as the final improved UNet3+ model.
[0009] Furthermore, a fractal curve module is introduced into the encoding stage of the improved UNet3+ model, so that the model can reorder image pixels and enhance pixel clustering, thereby improving intra-class compactness and inter-class separability.
[0010] Furthermore, the improved UNet3+ model adopts the adhesion coefficient measurement method to evaluate the performance of the model in edge segmentation. This indicator can more accurately quantify the improvement of the model in edge segmentation and provide targeted guidance for model optimization.
[0011] Furthermore, the improved UNet3+ model optimizes the network structure and parameter quantity, and the model achieves lightweight while maintaining high performance.
[0012] In summary, the technical effects and advantages of the present invention are as follows:
[0013] In the present invention, by introducing a fractal curve module and bilinear interpolation technology in the encoder stage, and introducing a convergence layer in the decoder part to select multi-dimensional feature maps of different levels, the model's ability to capture edge features is significantly improved, the accuracy of feature extraction is enhanced, and the segmentation quality is improved, making it more suitable for processing complex scenes and subtle boundaries of objects in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below.
[0015] Figure 1 This is a framework diagram of the improved UNet3+ model in the embodiment of the present application;
[0016] Figure 2 This is a bilinear interpolation calculation diagram in the embodiment of this application;
[0017] Figure 3 This is a comparison diagram before and after the Hilbert curve acts on the digital template in the embodiment of the present application;
[0018] Figure 4 This is a processing diagram of the convergence layer in the embodiment of the present application;
[0019] Figure 5 This is an example of a WHDLD data set and an RGB correspondence table in an embodiment of the present application;
[0020] Figure 6This is a comparison diagram of WHDLD segmentation results in the embodiment of this application;
[0021] Figure 7 This is a diagram of the WHDLD data set experiment results in the embodiments of this application. DETAILED DESCRIPTION
[0022] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0023] Example: Reference Figure 1-7 A remote sensing image semantic segmentation method based on fractal curves is shown, comprising the following steps: in the encoding stage of the UNet3+ model, the model uses a bilinear interpolation layer to replace the traditional pooling operation. This interpolation method can retain spatial information and reduce the loss of details during the downsampling process. At the same time, the fractal curve module is introduced to enable the model to reorder image pixels, enhance pixel aggregation, and thus improve intra-class compactness and inter-class separability;
[0024] Bilinear interpolation is a method for interpolation in two-dimensional space. It considers the nearest neighbor points around the sampling point and performs weighted average of the values of these points according to the relative distance. This method can not only retain more spatial information, but also smooth the image to a certain extent, reduce the appearance of jagged edges, and has strong adaptability to features of different scales.
[0025] Specifically, for each pixel after downsampling, we consider the four nearest neighboring points around its corresponding position in the original image. These four points form a rectangular area, and the target point is located in this rectangle. We first perform linear interpolation in the horizontal direction, and then perform linear interpolation again in the vertical direction to obtain the final interpolation result. The calculation process is as follows Figure 2 As shown, we can intuitively see the positional relationship between the target point and the four surrounding points. The mathematical expression is shown in the formula;
[0026] f(P)=(x2-x)(y2-y)f(Q 11 )+(x-x1)(y2-y)f(Q 21 )
[0027] +(x2-x)(y-y1)f(Q 12 )+(x-x1)(y-y1)f(Q 22 )
[0028] The encoder stage also uses a fractal curve module, and the pixel wandering paths generated by Hilbert Curves of different orders m are suitable for the specification of 2 mAfter weighing, we choose m as 8 for the experiment, and the final research size of the image is determined to be 256px. Algorithm 1 summarizes this operation step.
[0029]
[0030]
[0031] Large-sized remote sensing images and mask images need to be cut into equal-sized square slices. In order to show the effect of fractal curves on image slices, a 4-order walking curve is given to act on 16 square slices. The construction method is as follows: a 16-square digital template is constructed through the designed matrix, and then a 4-order Hilbert curve is applied to the digital template, which will generate a pixel chain with a length of 256. The generated pixel chain is then reconstructed to the size of the original image. After observation, it is found that the originally relatively discrete points are straightened and reconstructed into similar pixel blocks that will be arranged in several rows, increasing the pixel aggregation effect and reducing the pixel edge adhesion problem of the original slice, thereby significantly improving the boundary line of the object in the image. At the same time, this effect of the fractal curve has a better effect on large-area image objects. The comparison before and after the Hilbert curve acts on the digital template is shown below. Figure 3 As shown;
[0032] In the decoding stage of the UNet3+ model, a multi-scale feature fusion strategy is used to effectively integrate features at different levels through the pooling layer. The design of this pooling layer takes into account the importance differences of features at different levels. By selectively fusing feature information, the efficiency of feature utilization is improved, thereby obtaining an improved UNet3+ model. Compared with the traditional UNet and UNet3+ models, this feature fusion mechanism of the improved UNet3+ model is more sophisticated and efficient, and is particularly suitable for processing complex scenes and subtle boundaries of objects in remote sensing images.
[0033] The traditional UNet architecture uses skip connections to directly concatenate feature layers of the same level in the encoder and decoder, while UNet3+ reduces the feature maps of different layers to the same dimension and then fuses them. However, these methods do not fully consider the importance differences of the semantic information contained in feature maps of different layers. In order to more effectively utilize the image feature layer information, the improved UNet3+ model proposes a feature selection and fusion strategy, which we call the G layer. The core idea of this method is to not only consider the concatenation of features of the same level, but also screen and integrate features from different layers. The specific implementation process is as follows: Figure 4 As shown in the figure, the workflow of the G layer is feature merging, sorting selection, and same-layer splicing. The advantage of this method is that it maximizes the feature information of different layers by selectively fusing them, uses the L1 norm to ensure that the feature layer selection is representative, reduces the computational complexity and the amount of training parameters of the decoder part, and achieves the goal of lightweight.
[0034] By implementing the G layer, the improved UNet3+ model can effectively integrate features at different levels while retaining important semantic information. This method is particularly suitable for processing complex scenes and subtle boundaries in remote sensing images. Compared with traditional UNet and UNet3+, the G layer of the improved UNet3+ model provides a more sophisticated and efficient feature fusion mechanism;
[0035] Acquire remote sensing image data and construct a remote sensing image dataset, divide it into training set and validation set in proportion, and use the improved UNet3+ model to predict and classify the remote sensing image data;
[0036] The training samples are used as input data of the improved UNet3+ network to train the improved UNet model. During the training process, the model with the best effect on the validation set is taken as the final improved UNet3+ model.
[0037] Among them, the improved UNet3+ model adopts the adhesion coefficient measurement method to evaluate the performance of the model in edge segmentation. This indicator can more accurately quantify the improvement of the model in edge segmentation and provide targeted guidance for model optimization.
[0038] The improved UNet3+ model optimizes the network structure and parameter quantity, and the model is lightweight while maintaining high performance, making it easy to deploy and apply in resource-constrained environments. This is particularly important for the practical application of remote sensing image processing, because it enables the model to be more widely used in traffic analysis, urban planning, disaster monitoring and other fields, providing more accurate and efficient image segmentation solutions.
[0039] Specific experiments
[0040] The WHDLD dataset is a collection of high-resolution remote sensing images acquired by satellites. The dataset divides objects into six categories: buildings, roads, sidewalks, vegetation, bare soil, and water bodies. This classification covers the main types of objects in urban environments and provides a comprehensive sample for urban remote sensing research. Since the sizes of the original images and mask images already meet the research requirements, they can be used directly without additional preprocessing steps. Figure 5 The images and masks of the dataset and the RGB color table corresponding to each category are displayed. This visualization not only intuitively displays the content of the dataset, but also provides a clear reference for subsequent analysis and processing.
[0041] In order to evaluate the semantic segmentation quality of remote sensing images, a new evaluation index is proposed to evaluate the object boundary adhesion coefficient of semantic segmentation. The lower the value, the better the model effect. The formula is as follows:
[0042]
[0043] The ablation experiment verifies that the module based on fractal curves is better than the original model. 2 Net, UNeXt and GCtx-UNet, Figure 7 This is the WHDLD dataset experimental effect diagram. Figure 6 The figure contains the segmentation effects of multiple models. (a) is the original remote sensing image, (b) is the corresponding label mask, (c) and (d) show the segmentation images of UNet3+ before and after adding the fractal curve module, and (e) and (f) show the UNet3+ segmentation images. 2 Net is added before and after the segmentation map, (g) is the segmentation map of the UNeXt network, (h) is the segmentation map of the GCtx-UNet network, and (i) shows the segmentation map of the improved UNet3+ model we proposed.
[0044] from Figure 6 The performance differences of different models can be observed in the segmentation results. The segmentation results of the UNet3+ network (column c) show that both Road and Pavement, as well as Road and Baresoil, are easily misclassified, and large-scale errors occur, such as misclassifying bare soil and roads as shrubs represented by green. This is mainly because the downsampling feature layer points of the UNet3+ model in the encoding stage are too discrete, resulting in weak model expression ability. In contrast, the model that incorporates the fractal curve module (column d) has significantly improved classification results in large areas. This is due to the rearrangement of the pixels of the multi-channel feature map in the deep layer, which makes the same type of pixels more clustered. The same improvement can also be seen in the UNet3+ model. 2 Net model, it was observed that the results after incorporating the fractal curve module (column f) were better than those without incorporation (column e), which reduced the probability of bare soil being misclassified as shrubs.
[0045] By comparison Figure 6 From the (c), (e), (g), (h), and (i) columns, we can clearly see that the segmentation effect of the improved UNet3+ model is better than that of the other four models. The improved UNet3+ model makes the pixels of the multi-channel feature map more clustered, increasing the aggregation of the same type. At the same time, the convergence layer effectively utilizes the features and fractal feature information of different scales to improve the segmentation accuracy of multi-classification of remote sensing images. Compared with other comparison models, the improved UNet3+ model has clearer and smoother segmentation edges when processing large-area objects, significantly reducing the mis-segmentation caused by interference from cluttered backgrounds, shadows, etc., as well as the boundary adhesion caused by the close distance between objects. Overall, the segmentation results of the improved UNet3+ model are better than those of UNet3+, U 2Net, UNeXt, and GCtx-UNet models. These qualitative observations are also supported by quantitative performance indicators, as shown in the table below:
[0046] Table 1 WHDLD dataset model experimental data
[0047]
[0048]
[0049] The effectiveness of the fractal curve module was verified through ablation experiments on the first four models in the table in pairs. The maximum values of MIoU, MPA, F1-score and adhesion coefficient all came from the model with the fractal curve module added. The evaluation indicators reached 52.48%, 71.12%, 65.92% and 3.72% respectively, which effectively proved the effectiveness of the fractal curve module. At the same time, the experimental results of the improved UNet3+ model that integrates the three ideas are better than other networks. MIoU, MPA, F1-score and adhesion coefficient reached 54.71%, 74.43%, 67.57% and 3.45% respectively, verifying the performance of the model. Figure 7 It can be seen that the pixel segmentation accuracy of Vegetation and Water is higher, while the pixel segmentation effect of Road and Bare soil with smaller area is relatively poor.
[0050] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A remote sensing image semantic segmentation method based on fractal curves, characterized in that: The following steps are involved: In the encoding stage of the UNet3+ model, the model uses a bilinear interpolation layer to replace the traditional pooling operation. This interpolation method can retain spatial information during the downsampling process and reduce the loss of details. In the decoding stage of the UNet3+ model, a multi-scale feature fusion strategy is used to effectively integrate features at different levels through the pooling layer. The design of this pooling layer takes into account the importance differences of features at different levels and improves the efficiency of feature utilization by selectively fusing feature information, thereby obtaining an improved UNet3+ model. Acquire remote sensing image data and construct a remote sensing image dataset, divide it into training set and validation set in proportion, and use the improved UNet3+ model to predict and classify the remote sensing image data; The training samples are used as input data of the improved UNet3+ network to train the improved UNet3+ model. During the training process, the model with the best effect on the validation set is taken as the final improved UNet3+ model.
2. The remote sensing image semantic segmentation method based on fractal curve according to claim 1, characterized in that: The fractal curve module is introduced into the encoding stage of the improved UNet3+ model, so that the model can reorder image pixels and enhance pixel aggregation.
3. The remote sensing image semantic segmentation method based on fractal curve according to claim 1, characterized in that: The improved UNet3+ model uses the adhesion coefficient metric to evaluate the performance of the model in edge segmentation.
4. The remote sensing image semantic segmentation method based on fractal curve according to claim 1, characterized in that: The improved UNet3+ model can make the model lightweight by optimizing the network structure and parameter quantity.