Deep hierarchical multi-semantic data processing method based on dynamic cobweb and loss function cooperation

Abstract

The invention discloses a deep hierarchical multi-semantic data processing method based on cooperation of a dynamic cobweb and a loss function, which can effectively recover the spatial resolution and detail information of an image and improve the precision and robustness of a segmentation result. The method comprises the steps that feature information of different scales is captured through a multi-scale residual block, a cobweb structure is adopted to represent a potential hierarchical dependency relationship between semantic categories, and weights are associated through matrix formalization; edge information of the detail part is extracted and enhanced through an edge enhancement module, and the recognition capability of a target boundary is improved; global context information of different scales is captured through hole convolution kernels of different sizes in an ASPP module, and the global feature capability of the model is improved; performing layer-by-layer spatial resolution recovery on the feature map by using a multi-scale residual block to extract feature information, and splicing the feature information with edge features from an encoder through an attention gate; and cooperatively adjusting the cobweb loss by using the optimized loss function after the dynamic adjustment is defined.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of remote sensing image semantic segmentation, and in particular to a deep hierarchical multi-semantic data processing method in which a dynamic spider web and a loss function collaborate. Background Art

[0002] Semantic segmentation aims to assign semantic labels to pixels and is crucial in remote sensing image interpretation. Most segmentation deep neural networks are designed with the assumption that target categories have no interaction and coordination, which leads to the accumulation of segmentation errors and fails to capture the structural essence of the visual world.

[0003] Existing semantic segmentation research has limitations in integrating semantic hierarchical structures. For example, methods for natural scenes are not applicable to remote sensing images, or the framework modification is complex, and most of them do not fully consider label dependencies and interactions. In the field of remote sensing, capturing and representing structured information is crucial to adapting to new environments and multi-level abstract understanding.

[0004] Remote sensing images have complex data attributes, such as heterogeneity, class imbalance, overlapping class conditional distribution, etc., and imaging conditions can cause photographic distortion, scale changes, and illumination changes, reducing class separability. Traditional hierarchical semantic segmentation methods face challenges in remote sensing images. The affiliation between classes is irreversible, and relying only on a single semantic level division can easily lead to inaccurate segmentation. Summary of the invention

[0005] In order to overcome the above technical defects, a method for processing deep hierarchical multi-semantic data by cooperating with a dynamic spider web and a loss function is provided in the embodiment of the present application. The specific scheme is as follows:

[0006] The deep hierarchical multi-semantic data processing method is applicable to the hierarchical semantic segmentation network HAG Net model built based on the ResUNet network with an encoder-decoder structure. The encoder part includes a multi-scale residual block and an edge enhancement module, the bridge part uses a common convolutional network and a dilated spatial pyramid pooling ASPP module, and the decoder part consists of a multi-scale residual block and an attention gate.

[0007] The deep layered multi-semantic data processing method includes:

[0008] In the encoder part, multi-scale residual blocks are used to capture feature information of different scales, and a spider web structure is used to represent the potential hierarchical dependencies between semantic categories. The associated weights are formalized through matrices. The edge enhancement module is used to enhance the edge information of the details and improve the recognition ability of the target boundary.

[0009] In the bridge part, the different-sized atrous convolution kernels in the ASPP module are used to capture global context information of different scales and improve the global feature capability of the model.

[0010] In the decoder part, a multi-scale residual block is used to restore the spatial resolution of the feature map layer by layer to extract feature information, and then concatenate it with the edge features from the encoder through an attention gate;

[0011] The dynamically adjusted optimization loss function is defined to coordinate the cobweb loss. The optimization loss function is constructed based on the basic loss function and pre-calculated association weights. The association weight matrix reflects the hierarchical dependency between semantic categories and is used to balance the weights of different semantic categories in the segmentation process.

[0012] Optionally, in a possible implementation, the matrix is ​​denoted by M cf , the associated weight is recorded as W cf , through the matrix M cf To formalize the association weight W cf ;

[0013] Where each element of the matrix W cf represents the association weight between the coarse category c and the detailed category f;

[0014] Association weight W cf The calculation is based on pixel matching statistics: W cf =M cf [c, f], where M cf [c, f] represents the number of pixel matches between category c and category f, which is accumulated by traversing all images and their corresponding labels.

[0015] Optionally, in one possible implementation, the optimization loss function is denoted by L cobweb , the basic loss function is recorded as L base , the dynamic factor is recorded as δ, and the optimization loss function is expressed as: L cobweb =L base +δ×(1-R), where the relationship ratio R is a quantitative value that quantifies the degree of dependence between semantic categories;

[0016] In the image semantic segmentation task, the optimal segmentation result H*(I) is expressed as the optimal label set for each pixel i in image I. in By optimizing the loss function L cobweb The optimal label obtained to minimize the loss function;

[0017] The optimization process of the image semantic segmentation task is expressed as:

[0018]

[0019] Optionally, in a possible implementation, when predicting from the subdivided category f to the coarse category c, the relationship ratio R of category c to category f iscf The calculation method is as follows: W cf =M cf [c,f], where M c，f is an element in the mapping matrix from subdivided category f to coarse category c, indicating the mapping strength of category f to category c. The denominator is for a given coarse category C, where F represents the total number of subdivided categories, ensuring that the relationship ratio is normalized;

[0020] At this point, the basic loss function and the adjusted fine category loss function are:

[0021] and Where N is the total number of pixels in the image, is the model predicting that the nth pixel belongs to its true category y n The probability of y n is the true category index of the pixel, belonging to the subdivided category set, and ∈ is a very small constant used to ensure numerical stability.

[0022] Optionally, in a possible implementation, when predicting from a coarse-classified category c to a detailed category f, the relationship ratio R of category f to category c is fc The calculation method is as follows:

[0023] W cf =M cf [c,f]; where M c，f is an element in the mapping matrix from subdivided category f to coarse category c, indicating the mapping strength of subdivided category f to coarse category c, and the denominator is for a given subdivided category f, where c represents the total number of coarse categories;

[0024] At this point, the basic loss function and the adjusted fine category loss function are:

[0025] and, Where N is the total number of pixels in the image, is the model predicting that the nth pixel belongs to its true category y n The probability of y n is the true category index of the pixel, belonging to the coarse classification set, ∈ is a very small constant used to ensure numerical stability.

[0026] Optionally, in a possible implementation, the number of coarse categories is 6 and the number of fine categories is 16.

[0027] Optionally, in a possible implementation, in the encoder part, the multi-scale residual block includes three parallel convolution paths, 3x3, 5x5 and 7x7 convolutions, which capture feature information of different scales through multi-resolution paths to enhance the overall performance.

[0028] Optionally, in a possible implementation, the number of edge enhancement modules is 4; the edge enhancement module is obtained by embedding a lightweight attention ECA module after a common residual block and an edge extraction convolution block;

[0029] In the decoding part, a multi-scale residual block is used to restore the spatial resolution of the feature map layer by layer, which goes through five decoding steps in total;

[0030] In the decoding process of each layer, the current feature map is concatenated with the corresponding skip connection feature map and enhanced by the attention gate to highlight the important feature areas;

[0031] Each layer of the decoder will also concatenate the feature map of the current layer with the edge features from the encoder and the feature information of the multi-scale residual block MultiRes Block, thereby further fusing the multi-scale feature information.

[0032] The above technical solution can achieve the following technical effects in the embodiments of the present application:

[0033] The boundary enhancement module aims to strengthen the edge information of feature extraction, thereby improving the model's ability to recognize target boundaries;

[0034] The multi-scale residual block captures multi-level features in the image through convolution operations of different scales, enhancing the expressiveness of the features.

[0035] The ASPP module can capture global context information of different scales through dilated convolution kernels of different sizes, further improving the global feature extraction capability of the model.

[0036] The decoder uniformly uses MultiRes Block to restore spatial resolution layer by layer, and a total of five decoding steps are performed. During the decoding process of each layer, the current feature map is spliced ​​with the corresponding skip connection feature map and enhanced by Attention Gate (AG) to highlight important feature areas. Each layer of the decoder also splices the feature map of the current layer with the edge features from the encoder and the feature information of MultiRes Block to further integrate multi-scale feature information. This structural design ensures that the decoder can effectively restore the spatial resolution and detail information of the image under complex and changeable input data conditions, thereby improving the accuracy and robustness of the segmentation results.

[0037] By using a non-tree structure (such as a spider web structure) to represent the relationship between categories, and using a category transfer matrix to formalize this relationship, where the matrix elements represent the association weights between the coarse category c and the detailed category f, and the association weights are calculated based on pixel matching statistics. This approach can better capture the hierarchical dependencies between categories and find the most appropriate label for each pixel in the image under the framework of hierarchical dependencies and dynamic loss adjustment.

[0038] Adjust the loss according to the category relationship: quantify the degree of dependence by calculating the relationship ratio between categories. When predicting from a detailed category to a coarse category, the adjusted detailed category loss function is obtained by adjusting the relationship ratio; when predicting from a coarse category to a detailed category, the adjusted coarse category loss function is obtained by adjusting the relationship ratio. This method of dynamically adjusting the loss function according to the category relationship can better adapt to the complex relationship between different categories and improve the performance of the model in multi-category segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0040] Figure 1 A flowchart of a deep hierarchical multi-semantic data processing method in which a dynamic spider web and a loss function collaborate in this application;

[0041] Figure 2 A schematic diagram of the network structure of the HAG Net model in this application;

[0042] Figure 3 For this application Schematic diagram of the performance of traditional and advanced networks on the GID test set;

[0043] Figure 4 This is a schematic diagram of the performance of traditional and advanced networks on the Shengtengcup test set;

[0044] Figure 5 This is a schematic diagram of the display on the Gid and shengtengcup test sets before and after adding Cobweb loss. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0046] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0047] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0048] Methods based on deep learning are widely used for semantic segmentation, but simply deepening the network by increasing the number of layers will face challenges, such as deep networks relying on small local receptive fields, which limits the ability to understand global context. Scholars have designed a variety of solutions, such as expanding the receptive field, building a pyramid structure, using edge perception or boundary constraints, using encoder-decoder architecture, introducing attention mechanisms, and using incremental learning. In the field of remote sensing image processing, traditional segmentation methods assign each pixel to a single category label, which cannot effectively express complex relationships and weakens the understanding of the structural integrity of the ground object.

[0049] Convolutional networks can automatically learn hierarchical feature representations. Common traditional methods include using Markov random fields and conditional random fields and designing cascade classifiers, but their ability to obtain contextual information at multiple scales is limited. Multiscale processing is important for retaining image context information, and multiscale context information can be integrated through a variety of structures. As the resolution of satellite images increases, semantic-level analysis algorithms are gradually developing, but there is a semantic gap. Commonly used coarse-to-fine hierarchical methods include a recursive coarse-to-fine framework based on image-level category labels, the introduction of Bayesian technology into hierarchical classification, knowledge extraction constraints, etc., and segmentation can also be performed through methods such as saliency maps and hierarchical region trees, but these methods have problems such as sensitivity to parameter selection, dependence on initial label accuracy, and high computational complexity.

[0050] In response to the above-mentioned hierarchical semantic segmentation problem, a new hierarchical semantic segmentation network (Hierarchical AttentionGate-MultiResUNet, HAG Net) model is proposed in the embodiment of the present application, which uses structured class relationships and dynamic factors to adjust the loss function to improve model performance. Different from the previous structured segmentation model that focuses on complex network design, the hierarchical semantic segmentation method is formalized as a pixel-based multi-label classification task, the reasoning relationship makes the hierarchy reflect the potential relationship between the categories, the segmentation is forced to obey the hierarchical structure, and the category hierarchy knowledge is encoded into the pixel embedding space.

[0051] First, in order to facilitate understanding of the technical solutions in the embodiments of the present application, the technical terms involved in the solutions are explained below:

[0052] 1.ResUNet network:

[0053] ResUNet is a deep learning model that combines the ResNet and U-Net structures and is often used for image segmentation tasks. It combines the encoder-decoder structure of U-Net and the residual connection characteristics of ResNet, enabling it to effectively propagate gradients in deep networks, alleviate the gradient vanishing problem, and improve the stability and performance of the model.

[0054] 2. Atrous Spatial Pyramid Pooling (ASPP) module:

[0055] The ASPP module is a technology used to expand the receptive field of convolution and can effectively capture multi-scale contextual information. By using hole convolutions with different sampling rates to process input feature maps in parallel, ASPP can enhance the model's ability to detect objects of different scales in images, and is particularly suitable for image segmentation tasks in complex scenes.

[0056] 3. Multi-scale residual block (MultiRes Block):

[0057] The Multi-scale Residual Block is an improved residual module designed to capture features at multiple resolutions simultaneously. By combining convolution kernels of different sizes in one module, the MultiRes Block can improve the diversity and accuracy of feature extraction while maintaining residual connections, making the model more advantageous when processing images with rich details.

[0058] 4. Skip connection feature map:

[0059] Skip connection is a widely used structure in U-Net and other networks to transfer feature maps between encoder and decoder. It helps preserve the original input features by directly connecting input and output between different network layers, thereby enhancing the ability to restore contextual information in the decoder stage and improving the accuracy of segmentation results.

[0060] Secondly, in order to facilitate those skilled in the art to understand the technical solutions in the embodiments of the present application, the HAG Net model and its deep hierarchical multi-semantic data processing method proposed in the present application are described below, as follows:

[0061] Embodiment 1

[0062] like Figure 1 As shown, the deep hierarchical multi-semantic data processing method in the embodiment of the present application is applicable to the hierarchical semantic segmentation network HAG Net model constructed based on the ResUNet network with a codec structure, wherein the encoder part includes a multi-scale residual block and an edge enhancement module, the bridge part adopts an ordinary convolutional network and a dilated spatial pyramid pooling ASPP module, and the decoder part consists of a multi-scale residual block and an attention gate.

[0063] The deep layered multi-semantic data processing method includes:

[0064] S1. In the encoder part, multi-scale residual blocks are used to capture feature information of different scales, and a spider web structure is used to represent the potential hierarchical dependencies between semantic categories. The associated weights are formalized through matrices. The edge enhancement module is used to lift and strengthen the edge information of the details, thereby improving the ability to recognize the target boundary.

[0065] Compared with ordinary residual blocks, the MultiRes Block captures feature information of different scales through multi-resolution paths, enhancing the feature extraction capability. The MultiRes Block is applied to the main network architecture to improve the overall performance of the model by capturing multi-scale features.

[0066] The edge enhancement module uses a normal residual block to improve the network's ability to capture edge information. Optionally, the number of edge enhancement modules is 4. The edge enhancement module is obtained by embedding a lightweight attention (Efficient Channel Attention, ECA) module after the normal residual block and the edge extraction convolution block.

[0067] Optionally, in some embodiments of the present application, in the encoder part, the multi-scale residual block contains three parallel convolution paths: the first path is 3x3 convolution, the second path is 5x5 convolution, and the third path is 7x7 convolution, where the filter allocations are int(w*0.167), int(w*0.333), and int(w*0.5). Through this design, the network can capture feature information of different scales at the same time, enhancing the understanding of image details.

[0068] Specifically, the matrix is ​​denoted as M cf , the associated weight is recorded as W cf , through the matrix M cf To formalize the association weight W cf , where each element of the matrix W cf represents the association weight between the coarse category c and the detailed category f; the association weight W cf The calculation is based on pixel matching statistics: W cf =M cf [c, f], where M cf [c, f] represents the number of pixel matches between category c and category f, which is accumulated by traversing all images and their corresponding labels.

[0069] Optionally, in some embodiments, the number of coarse categories is 6 and the number of fine categories is 16.

[0070] S2. In the bridge part, the different sizes of hole convolution kernels in the ASPP module are used to capture global context information of different scales and improve the global feature capability of the model.

[0071] The bridge part uses a common convolutional network and a dilated spatial pyramid pooling ASPP module. The ASPP module can capture global context information of different scales through dilated convolution kernels of different sizes, further improving the global feature extraction capability of the model.

[0072] S3. In the decoder part, a multi-scale residual block is used to restore the spatial resolution of the feature map layer by layer to extract feature information, and then it is spliced ​​with the edge features from the encoder through the attention gate.

[0073] The decoder uniformly uses the multi-scale residual module MultiRes Block to restore the spatial resolution layer by layer, and a total of five decoding steps are performed. During the decoding process of each layer, the current feature map is spliced ​​with the corresponding skip connection feature map and enhanced by the attention gate (AG) to highlight the important feature areas. Each layer of the decoder will also splice the feature map of the current layer with the edge features from the encoder and the feature information of the MultiRes Block to further integrate the multi-scale feature information. This structural design ensures that the decoder can effectively restore the spatial resolution and detail information of the image under complex and changeable input data conditions, thereby improving the accuracy and robustness of the segmentation results.

[0074] The two inputs of the attention gate module are skip_connection, gating_signal, W x and W g The main structure consists of a 1×1 convolution and an activation function. Finally, the attention weight map is upsampled (upConv_Block) and deconvolved (trans_conv2D) to ensure that its size matches skip_connection. By dynamically adjusting the weights of features through the gating unit, the gated attention mechanism can not only effectively enhance the attention to multiple targets, but also maintain the balance of attention between categories in multi-target scenarios, thereby significantly improving the overall model performance. Through this method of dynamically adjusting feature weights, the gated attention mechanism can maintain high accuracy and performance when processing complex multi-target tasks.

[0075] S4. Use the dynamically adjusted optimization loss function to coordinately adjust the spider web loss.

[0076] Among them, the optimization loss function is constructed based on the basic loss function, pre-calculated association weights and dynamic factors. The dynamic factor can adjust the optimization loss function according to the average loss of multiple cycles adjacent to the current training cycle. The association weight matrix reflects the hierarchical dependency between semantic categories and is used to balance the weights of different semantic categories in the segmentation process.

[0077] Optionally, in some embodiments, the optimization loss function is denoted by L cobweb , the basic loss function is recorded as L base , the dynamic factor is recorded as δ, and the optimization loss function is expressed as: L cobweb =L base +δ×(1-R), where the relationship ratio R is a quantitative value that quantifies the degree of dependence between semantic categories;

[0078] In the image semantic segmentation task, the optimal segmentation result H*(I) is expressed as the optimal label set for each pixel i in image I. in By optimizing the loss function L cobweb The optimal label obtained to minimize the loss function;

[0079] The optimization process of the image semantic segmentation task is expressed as:

[0080]

[0081] Furthermore, when predicting from the subdivided category f to the coarse category c, the relationship ratio R of category c to category f cf The calculation method is as follows: W cf =M cf [c,f], where M c，f is an element in the mapping matrix from subdivided category f to coarse category c, indicating the mapping strength of category f to category c. The denominator is for a given coarse category C, where F represents the total number of subdivided categories, ensuring that the relationship ratio is normalized;

[0082] At this point, the basic loss function and the adjusted fine category loss function are:

[0083] and Where N is the total number of pixels in the image, is the model predicting that the nth pixel belongs to its true category y n The probability of y n is the true category index of the pixel, belonging to the subdivided category set, and ∈ is a very small constant used to ensure numerical stability.

[0084] Furthermore, when predicting from the coarse category c to the fine category f, the relationship ratio R of category f to category c is fc The calculation method is as follows:

[0085] W cf =M cf [c,f]; where M c，f is an element in the mapping matrix from subdivided category f to coarse category c, indicating the mapping strength of subdivided category f to coarse category c, and the denominator is for a given subdivided category f, where C represents the total number of coarse categories;

[0086] At this point, the basic loss function and the adjusted fine category loss function are:

[0087] and, Where N is the total number of pixels in the image, is the model predicting that the nth pixel belongs to its true category y n The probability of y n is the true category index of the pixel, belonging to the coarse classification set, ∈ is a very small constant used to ensure numerical stability.

[0088] Finally, the update of the dynamic factor δ is based on the average loss of the past 5 training cycles and is calculated as follows:

[0089] In the technical solution of this application, the improvement of using optimized loss function to coordinately adjust spider web loss is reflected in the following aspects:

[0090] 1. Consider class hierarchical dependencies

[0091] Limitations of traditional methods: Traditional segmentation methods often do not fully consider the hierarchical dependencies between categories when dealing with semantic segmentation tasks. For example, in single-layer semantic segmentation, each pixel is simply classified into a predefined semantic category without considering the association between categories. This method cannot accurately capture the structural essence of the visual world when processing complex remote sensing image data, resulting in accumulated segmentation errors.

[0092] Improvement of the dynamic spider web model: By adopting a non-tree structure (such as a spider web structure) to represent the relationship between categories, and using a category transfer matrix to formalize this relationship, where the matrix elements represent the association weights between the coarse category c and the detailed category f, and the association weights are calculated based on pixel matching statistics. This method can better capture the hierarchical dependencies between categories and find the most appropriate label for each pixel in the image under the framework of hierarchical dependencies and dynamic loss adjustment.

[0093] 2. Dynamically adjust the loss function

[0094] Disadvantages of traditional loss functions:

[0095] Traditional loss functions (such as cross entropy loss sparse_categorical_crossentropy) only consider the difference between the predicted value and the true value of the pixel itself, which can easily lead to a decrease in model performance when dealing with complex category imbalance and multi-category segmentation problems.

[0096] Adjust the loss according to the category relationship: quantify the degree of dependence by calculating the relationship ratio between categories. When predicting from a detailed category to a coarse category, the adjusted detailed category loss function is obtained by adjusting the relationship ratio; when predicting from a coarse category to a detailed category, the adjusted coarse category loss function is obtained by adjusting the relationship ratio. This method of dynamically adjusting the loss function according to the category relationship can better adapt to the complex relationship between different categories and improve the performance of the model in multi-category segmentation tasks.

[0097] Embodiment 2

[0098] The encoder-decoder structure is a simple and effective semantic segmentation structure. The encoder part gradually reduces the spatial dimension through the pooling layer to obtain features from shallow to deep. The decoder gradually restores the target details and spatial dimensions through deconvolution or interpolation. It also ensures that the output has the same dimension as the input image, thus achieving end-to-end training. There are usually shortcut connections from the encoder to the decoder to help the decoder better restore object details.

[0099] Simple encoder-decoder structures often lose global information by directly fusing the deepest features without any other possible operations. This leads to misclassification of pixels inside large buildings in dense classification of high-resolution aerial images.

[0100] Therefore, it is necessary to design a network that can utilize the stability of the codec structure and avoid losing global features. That is, the HAG Net model in this application is also a ResUNet network based on the codec structure. Its network structure is as follows: Figure 2 As shown:

[0101] The hierarchical semantic segmentation network HAG Net proposed in this application is mainly based on the ResUNet network. The encoder part consists of 4 boundary enhancement modules and multi-scale residual blocks. The boundary enhancement module is designed to enhance the edge information of feature extraction, thereby improving the model's ability to recognize the target boundary. The multi-scale residual block captures multi-level features in the image through convolution operations of different scales, enhancing the expression ability of the features. The bridge part uses a normal convolutional network and a hole space pyramid pooling ASPP module. The ASPP module can capture global context information of different scales through hole convolution kernels of different sizes, further improving the global feature extraction ability of the model. The decoder uniformly uses the multi-scale residual module MultiRes Block to restore the spatial resolution layer by layer, and a total of five decoding steps are performed. In the decoding process of each layer, the current feature map is spliced ​​with the corresponding skip connection feature map and enhanced by the attention gate (AG) to highlight the important feature area. Each layer of the decoder also splices the feature map of the current layer with the edge features from the encoder and the feature information of the MultiRes Block, thereby further fusing the multi-scale feature information. This structural design ensures that under complex and changeable input data conditions, the decoder can effectively restore the spatial resolution and detail information of the image, thereby improving the accuracy and robustness of the segmentation results.

[0102] Ordinary residual block and multi-scale residual block:

[0103] exist Figure 2 The HAG Net network structure shown in the figure mainly uses two types of residual blocks, one is the edge residual block used for edge enhancement, and the other is the multi-scale residual block used in the main network architecture. The edge residual block used in the edge enhancement module edge enhanced block is composed of basic residual blocks. Compared with ordinary residual blocks, the multi-resolution residual block (MultiRes Block) captures feature information of different scales through multi-resolution paths, thereby enhancing the feature extraction capability. The ordinary residual block solves the gradient vanishing problem in deep networks through shortcut connections, but is not as detailed and diverse as the multi-resolution block in feature extraction. The residual block used for edge enhancement is usually composed of basic residual blocks (basic block) to improve the network's ability to capture edge information; while the multi-scale residual block is applied to the main network architecture to improve the overall performance of the model by capturing multi-scale features. The multi-resolution residual block contains three parallel convolution paths: the first path is 3x3 convolution, the second path is 5x5 convolution, and the third path is 7x7 convolution, where the filter assignments are int(w*0.167), int(w*0.333), and int(w*0.5). Through this design, the network can simultaneously capture feature information of different scales and enhance the understanding of image details.

[0104] Edge Enhanced block:

[0105] When facing the task of fine-scale semantic segmentation at a low semantic level, the diversity and overlap of categories are common, and there are also complex situations where the same object has different spectra and different objects have the same spectra. These problems significantly increase the difficulty of the segmentation task, especially when dealing with weak boundaries. To overcome these challenges, we use the edge enhancement method corresponding to the ResUNet network and integrate it into the encoder part to improve the ability to extract edge features.

[0106] Specifically, we introduced residual blocks and edge extraction convolution blocks corresponding to multi-scale residual blocks (MultiResBlock) in the encoder part of ResUNet to capture feature information of different scales and enhance the ability to parse details. At the same time, by embedding a lightweight attention (EfficientChannelAttention, ECA) module after the ordinary residual block and edge extraction convolution block, the expressiveness of the feature map and the edge positioning accuracy are further improved. The input of the encoder first passes through the backbone network to extract features, and then passes through five residual blocks in sequence, with an edge enhancement EE block set after each residual block. The obtained boundary enhancement result is combined with the feature map of MultiResBlock.

[0107] These rich features are passed to the decoder through AttentionGate. AttentionGate plays an important role in this process. By dynamically adjusting the weights of feature maps, the network can pay more attention to key feature areas, thereby improving the accuracy and robustness of segmentation results. This design not only takes advantage of the edge enhancement method, but also combines the advantages of multi-scale feature extraction and attention mechanism to ensure that the model can maintain high segmentation accuracy and stability under complex and changing input data.

[0108] In this way, our model can effectively deal with weak boundary phenomena when processing fine-scale segmentation tasks of high-resolution images, significantly improving the overall segmentation performance. Combining multi-scale feature extraction, edge enhancement and attention mechanism, the model performs well in complex multi-target tasks, significantly improving segmentation accuracy and robustness. Through this design, the model not only performs well in high-resolution image processing, but also maintains stable segmentation effects in complex scenes, significantly improving the overall segmentation performance and wide application.

[0109] Attention-Gate:

[0110] Deep learning models trained with attention gates can implicitly improve network performance, and many studies focus on binary classification tasks such as single-category scenarios such as medical imaging and building extraction. Although AttentionGate can further enhance the performance of the model in multi-category tasks, it has not been widely used in remote sensing image segmentation. By dynamically adjusting the weights of features through the gating unit, the gated attention mechanism can not only effectively enhance the attention to multiple targets, but also maintain the balance of attention between categories in multi-target scenarios, thereby significantly improving the overall model performance. Through this method of dynamically adjusting feature weights, the gated attention mechanism can maintain high accuracy and performance when processing complex multi-target tasks. The two inputs of the Attention Gate module are skip_connection, gating_signal, W x and W g The main structure consists of a 1×1 convolution and an activation function. Finally, the attention weight map is upsampled (upConv_Block) and deconvolved (trans_conv2D) to ensure that its size matches skip_connection.

[0111] Segmentation network:

[0112] Different from typical segmentation methods that treat semantic classes as disjoint labels, in deep semantic segmentation of high-resolution remote sensing images, the potential dependencies between classes are considered and not only in the form of a tree-like hierarchy. In single-layer semantic segmentation, each pixel is classified into a predefined semantic category without considering the dependencies between categories. For an image I, the goal is to assign a label l to each pixel i. i , where l i ∈L, L is the set of all possible labels. Therefore, single-layer semantic segmentation can be expressed as:

[0113]

[0114] Here S(I) represents the result of single-layer semantic segmentation of image I, that is, the set of labels assigned to each pixel in the image.

[0115] In multi-layer (hierarchical) semantic segmentation, the hierarchical dependencies between labels are considered. Considering the hierarchical relationship between categories, we use a non-tree structure, such as a spider web structure, to represent this relationship. This structure is represented by the matrix M cf To formalize, where each element of the matrix W cf Represents the association weight between the coarse category c and the detailed category f. Association weight W cf The calculation of is based on pixel matching statistics:

[0116] W cf =M cf [c,f] (2)

[0117] Here, M cf [c, f] represents the number of pixel matches between category c and category f, which is accumulated by traversing all images and their corresponding labels. Then the multi-layer semantic segmentation can be expressed as: This statement clearly reflects the goal of multi-layer semantic segmentation, which is to find the most appropriate label for each pixel in the image under the framework of hierarchical dependency and dynamic loss adjustment.

[0118]

[0119] H * (I) represents the optimal result obtained by performing multi-layer semantic segmentation on image I while considering the hierarchical dependencies between categories and dynamic loss adjustment. is the optimal label for each pixel i obtained by solving the optimization problem.

[0120] Multiple losses synergistically regulate cobweb loss

[0121] In order to incorporate the hierarchical relationship between categories into the loss function, we define a dynamically adjusted loss L cobweb , which is based on the basic loss Lbase and the pre-computed association weight W cf and a dynamic factor δ, so as to fully consider the hierarchical dependencies between categories and the dynamic changes of the loss during training in the optimization process. I is a given image, where the goal of each pixel i is to be assigned a label l i Our goal is to find a label distribution L = {l1, l2, ..., l N}, so that given an image I, the association weight matrix W between categories cf and dynamic factor δ, the total loss is minimized. * Represents the optimal label assignment, that is, the final semantic segmentation result.

[0122] L cobweb It is a loss function that takes into account the hierarchical dependencies between categories. Its specific definition is as follows: L base The traditional loss function cross entropy loss sparse_categorical_crossentropy is used to measure the true label of pixel i and predicted labels The difference between cf The correlation ratio weight matrix between categories reflects the hierarchical dependency between categories by calculating the correlation between different categories. cf To formalize, where each element of the matrix W cf It represents the strength of the association weight between the coarse classification c and the detailed classification f, so that the label assignment depends not only on the predicted value of the pixel itself, but also on the association between the categories. δ is a factor dynamically calculated based on the average loss of the last few training cycles. Its function is to adjust the loss function in real time during the training process to balance the weights of different categories in the segmentation process and prevent the degradation of model performance due to category imbalance or fluctuations during training.

[0123] In the image semantic segmentation task, the optimal segmentation result H*(I) is expressed as the optimal label set for each pixel i in image I. in By optimizing the loss function L cobweb The optimization process aims to minimize the loss of each pixel, that is, by minimizing L cobweb To determine the optimal label assignment. Specifically, the optimal label is achieved by minimizing the loss of each pixel The optimization process can be expressed as formula (4). The core idea of ​​this formula is that for each pixel i, we find the label that minimizes the loss function in the possible label set L. Compared with the general formula (3) In contrast, the optimization process here explicitly selects the optimal label for each pixel by minimizing the loss function, rather than just a simple label assignment. Therefore, compared to traditional label assignment methods, our optimization process not only focuses on the prediction error of a single pixel, but also cf and δ to adjust the global strategy of label assignment, so that the final segmentation result can better capture the hierarchical dependencies between categories and improve the overall performance of the model

[0124]

[0125] We calculate the relationship ratio R of category c to category f to quantify the degree of dependence between categories:

[0126] L cobweb =L base +δ×(1-R) ​​(5)

[0127] When we go from detailed prediction (16 categories, represented by f) to coarse prediction (6 categories, represented by c), the relationship ratio R of category c to category f is cf It can be calculated as follows:

[0128] W cf =M cf [c,f] (6)

[0129]

[0130] Here M c，f It is an element in the mapping matrix from 16 categories to 6 categories, indicating the mapping strength from category f to category c. The denominator is for a given C (coarse classification), where F represents the total number of subdivided categories, ensuring that the relationship ratio is normalized. The loss function at this time is

[0131]

[0132]

[0133] N is the total number of pixels in the image. is the model predicting that the nth pixel belongs to its true category y n The probability of y n is the true category index of the pixel, belonging to the subdivided category set, and ∈ is a very small constant used to ensure numerical stability. is the adjusted fine category loss function, is the basic loss function.

[0134] When we go from coarse classification (6 categories, represented by c) to detailed classification (16 categories, represented by f), the relationship ratio R of category f to category c is fc It can be calculated as follows:

[0135]

[0136] The denominator is for a given subdivision category f, where C represents the total number of coarse categories. The loss function at this time is:

[0137]

[0138] N is the total number of pixels in the image. is the model predicting that the nth pixel belongs to its true category y n The probability of y n is the true category index of the pixel, belonging to the coarse classification set, ∈ is a very small constant used to ensure numerical stability.

[0139] Finally, the semantic stratification effect of the deep stratified multi-semantic data processing method in the embodiment of the present application is described using The GID Datasets as the basis for training and evaluation:

[0140] Dataset:

[0141] As the basis for training and evaluation, a well-annotated benchmark dataset is crucial for developing remote sensing semantic segmentation algorithms. It is difficult for remote sensing semantic segmentation datasets to balance the number of data samples and the adequacy of land cover categories. Therefore, the growing demands in real-world applications, such as simultaneous identification of industrial land and urban residential areas, are difficult to meet.

[0142] GID Datasets are selected as the experimental data set, including GID-5 and GID-15. GID-5 contains 150 images covering dozens of cities in China, with 5 land cover categories; GID-15 subdivides the categories of GID-5 and determines 15 relevant categories with reference to China's land use classification standards.

[0143] GID-5 contains rich data samples covering dozens of cities in China, 150 images of 6800×7200 pixels, which ensures the data diversity of deep model training, while only 5 land cover categories (i.e. built-up area, farmland, forest, meadow and water) are not enough for practical applications. The WHU GID-5 dataset is a large-scale, high-resolution remote sensing image land cover dataset based on China's Gaofen-2 satellite images. The dataset consists of 150 pixel-level annotated images, including five land cover categories: buildings, farmland, forest, grassland and water. Red is for buildings, green is for farmland, cyan is for forest, yellow is for meadow and blue is for water. The image size is 6800×7200, and each image includes four bands (i.e. blue, green, red and near-infrared bands).

[0144] In order to propose a better benchmark that meets actual needs, the land cover categories in GID-5 were subdivided to form a new GID-15. Specifically, in order to further distinguish subcategories belonging to the same general land cover category, the Chinese Land Use Classification Standard (GB / T21010-2017) was referenced, and 15 relevant categories were finally determined through hierarchical classification relationships. Specifically, in order to study the detailed distribution of urban areas, built-up areas were subdivided into industrial land, urban residential land, rural residential land, and transportation land. Then, forests were subdivided into garden land, arbor forest, and shrub land for vegetation coverage research, and meadows were subdivided into natural meadows and artificial meadows. Finally, farmland was subdivided into paddy fields, irrigated land, and dry land to study the distribution of agricultural land; water areas were subdivided into rivers, lakes, and ponds to study water resources.

[0145] The "Huawei Ascend Cup" remote sensing image dataset is a dataset provided in one of the tracks of the Huawei Ascend Cup competition on August 24, 2020. This dataset focuses on processing optical remote sensing images with typical land use classifications. Based on the existing remote sensing classification needs and actual needs, and referring to the geographical national conditions monitoring and the "Three National Land Surveys" land classification standards, a feature classification system that follows the "what you see is what you get" principle is designed. The dataset contains 8 main categories and 17 secondary subcategories. It should be noted that the main categories and secondary categories do not directly correspond to remote sensing images. In this study, the main categories used were generated based on the secondary subcategory labels defined in the official classification table.

[0146] Data source: This dataset is collected, annotated and constructed by Pengcheng Laboratory and its partners. The data sources include visible light and multispectral images from Gaofen-1, Gaofen-2, Gaofen-6, SuperView-2, Beijing-2 and some aerial images, with resolutions ranging from 0.1 meters to 4 meters.

[0147] Dataset size: The dataset includes more than 1 million semantic segmentation samples of remote sensing images.

[0148] Original image: The image is in TIFF format, containing red (R), green (G), and blue (B) bands. The training set image size is 256×256 pixels, the initial test set image size is also 256×256 pixels, and the subsequent rounds of test set image size can be changed.

[0149] Label data: The labels are stored in single-channel PNG files, with each pixel value represented by three digits in the storage format of 'uint16'. The hundreds digit represents the main category, while the tens and ones digits together represent the secondary category. A total of 100,000 main category images and 100,000 secondary subcategory images are provided. This study uses 16,000 images as a training set and 7,576 images as a validation set.

[0150] The data samples are selected from the GID-5, GID-15 datasets, Huawei Ascend Cup 2020-17 and Huawei Ascend Cup 2020-8.

[0151] Evaluation indicators:

[0152] The model performance is evaluated using indicators such as overall accuracy (OA), precision, recall, F1 score, and mean intersection over union (MIOU). These indicators have been widely used in previous studies and compared with recognized semantic segmentation evaluation standards to comprehensively evaluate the performance of the proposed model.

[0153] Specifically, the overall accuracy (OA) measures the proportion of correct predictions of the model among all samples; precision reflects the proportion of samples predicted as positive that are actually positive; recall represents the proportion of samples that are correctly predicted as positive among all samples that are actually positive; F1 score is the harmonic mean of precision and recall, which is used to measure the comprehensive performance of the model in the classification task; mean intersection over union (MIOU) is used to evaluate the segmentation performance of the model on multiple categories. These evaluation indicators are calculated as follows: P represents the number of positive samples; N represents the number of negative samples; TP is the number of correctly predicted positive samples (True Positives); FP is the number of samples incorrectly predicted as positive (False Positives); TN is the number of correctly predicted negative samples (True Negatives); FN is the number of samples incorrectly predicted as negative (False Negatives). Through the comprehensive evaluation of these indicators, we can fully understand the performance of the proposed model in different aspects, so as to more accurately judge its advantages and disadvantages:

[0154]

[0155] Parameter settings:

[0156] An Intel i9-7940X CPU with 8 cores and 16 threads, NVIDIA RTX3090 GPU×2 and 64GB memory are used. The software environment includes 64-bit Microsoft Windows 10 operating system, Anaconda 5.2.0, CUDA 11.2, Python3.8 and TensorFlow 2.5.0.

[0157] The transposed layer is initialized using the Xavier initialization method, and the Adam optimizer is used with a learning rate of 4.5×10 -4 , trained for 70 epochs with a batch size of 8.

[0158] Results and Discussion:

[0159] In this section, we demonstrate the performance of the proposed method using the GID and ISPRS Vaihingen datasets and analyze its effect.

[0160] HAGNet network performance:

[0161] In our proposed L cobweb In the loss function, the category transfer matrix is ​​used to effectively capture the specific relationship between categories and is dynamically designed into the loss function, forming a mandatory constraint between multiple labels. cobweb It has shown significant advantages in multiple models and indicators. Table 1 shows the classification ability of HAGNet in coarse category features. From the data, L cobweb The loss function in the Deeplabv3+ model is compared to L base The loss function is improved. For example, the IoU in the meadow category is increased to 77.05%, which is higher than L base At the same time, the mIoU and FWIoU indicators have also improved, reaching 88.85% and 87.83% respectively. These data reflect that L cobweb The performance on complex categories is more stable, and its dynamic adjustment mechanism can effectively handle the subtle differences in multi-category segmentation, avoiding the limitations of traditional single constraint criteria. cobweb It also significantly improves the performance of multiple categories, especially in the water category, where the IoU reaches 93.42%, which is much better than L base The 86.38% improvement is significant, demonstrating that L cobweb The advantage of L cobwebThe mIoU and FWIoU of the proposed model HAG are increased to 90.41% and 91.38% respectively, showing its robustness among different categories. In our proposed model HAG model, the IoU of farmland and water categories reached 97.26% and 96.06% respectively, which is higher than that of L base All of them have been improved, further verifying the effectiveness of its dynamic loss adjustment mechanism. cobweb Through the dynamic design of inference relations and category transfer matrices, the hierarchical structure is encoded into the pixel embedding space, ensuring high accuracy of the segmentation task in complex scenes.

[0162] In Table 2, we can see that L cobweb The loss function shows significant advantages in each model. base The only difference is that L cobweb The class transfer matrix is ​​used to capture the hierarchical relationship between categories and dynamically adjust the loss function, thereby achieving more refined constraints on multi-class segmentation tasks. This method not only improves the adaptability of the model to various scenarios, but also significantly improves the segmentation accuracy of different categories.

[0163] In the Deeplabv3+ model, L cobweb The classification error in complex scenes is reduced, and the overall accuracy is improved to 79.93%. In particular, the IoU performance in the Garden and River categories reaches 90.23% and 80.19% respectively, which is significantly better than L base method. This shows that L cobweb It can better capture the subtle differences between categories and enhance the model's ability to segment complex scenes. For the MAE-BG model, L cobweb The IoU in multiple categories has been significantly improved, for example, Urban land has reached 80.53% and Dry crop has reached 85.95%. base Method, L cobweb It more effectively handles the complex relationship between category levels, making the overall accuracy reach 83.39%, showing strong generalization ability, especially when dealing with multi-category segmentation tasks, and is more robust. cobweb The introduction of the loss function has enabled the model to achieve significant improvements in multiple categories, especially in the Pond category, where the IoU increased from 18.46% to 24.02%, which is particularly outstanding. This further shows that through the dynamic adjustment of the category transfer matrix, the model can better capture the complex dependencies between categories, effectively improving the performance of the model in various complex scenarios.

[0164] In short, L cobwebThe loss function significantly enhances the accuracy and robustness of the model when dealing with multi-category segmentation tasks by dynamically adjusting the hierarchical relationship between categories. base A simple superposition method, L cobweb It provides a more flexible and effective segmentation strategy, which is especially suitable for high-resolution remote sensing image segmentation tasks that need to process complex category associations.

[0165] Table 1 shows that HAG exhibits excellent performance in high semantic categories, especially in the building, farmland and grassland categories, showing superior performance and excellent segmentation ability. In the building category, HAG achieved a precision of 99.07 and a recall of 98.27, which is 1.51 precision and 1.08 recall higher than the worst segmentation network CMLF. In the farmland category, HAG has a precision of 96.87 and a recall of 97.64, which is 2.65 precision and 4.25 recall higher than the worst SUNet. For the meadow category, HAG performs best with a precision of 99.08 and a recall of 96.31, which is 1.91 precision and 1.25 recall higher than CGCL. According to Table 2, HAG achieves an F1 score of 98.66 and an IoU of 97.26 in the building category, surpassing the worst-performing SUNet by 0.76 F1 score and 1.37 IoU, respectively. In the farmland category, HAG achieves an F1 score of 96.57 and an IoU of 91.73, which are 2.77 F1 score and 15.84 IoU higher than SUNet, respectively. In the grass category, HAG achieves an F1 score of 98.17 and an IoU of 96.06, which are 2.40 F1 score and 4.17 IoU higher than SUNet. Compared with the best SFFNet, HAG is only 0.77 lower in IoU in the building category, but 0.88 higher in F1 score in the farmland category. This accuracy is not only better than the worst-performing network SUNet, but also 0.76 higher in F1 score and 1.37 higher in IoU. According to the experimental results in Tables 3 and 4, our method (HAG) shows excellent performance in the semantic segmentation of the GID dataset, especially showing significant advantages in some high semantic categories. In the urban land (Urb.land) category, HAG achieved an accuracy of 88.21 and an IoU of 77.9, which is 10.92 higher than the worst performing network SUNet by 11.19. In the traffic land (Traff.land) category, HAG achieved an accuracy of 78.5 and an IoU of 66.14, which is 55.42 higher than SUNet by 48.07, demonstrating its strong ability to capture complex traffic features. In the irrigated land (Irr.land) category, HAG achieved an accuracy of 91.21 and an IoU of 87.53, which is significantly higher than the poorly performing Deeplabv3+ by 12.4 higher than the poorly performing Deeplabv3+ by 15.88. In addition, in the natural grass (Nat.mead) category, HAG achieved an accuracy of 95.04% and an IoU of 68.03, which is 31.76% higher than SUNet and 17.01% higher than SUNet.Compared with the best performing SFFNet, HAG still shows competitiveness in some categories such as urban land and traffic land. In the categories of irrigated land and natural grassland, HAG's performance is slightly lower than SFFNet, but it still maintains strong performance in other categories. Overall, HAG shows excellent stability and accuracy in multi-category scenarios, especially in complex and high semantic categories, verifying its advantages as an efficient and reliable semantic segmentation method on the GID dataset. Figure 3 As shown in the figure, the segmentation effect of the GID test set in the above networks is shown. Deeplabv3+ relies on the atrous spatial pyramid pooling (ASPP) module, which can capture multi-scale context information well, but the segmentation results are slightly rough in areas with complex details. The CGGL network guides global context modeling through category information and performs stably, but the detail capture in some edge areas is slightly insufficient. The MAE-BG network effectively enhances the detail segmentation ability through edge detection and boundary guidance, and shows excellent edge continuity. Although SUNet combines the Swin Transformer and UNet architectures and has good context modeling capabilities, its segmentation accuracy in complex areas is not as ideal as other networks. CMLF shows strong feature extraction capabilities and detail retention, which is suitable for segmentation of large areas, but there is still room for improvement in edge processing. SFFNet makes full use of spatial information, performs well in complex areas and edge processing, and maintains a high segmentation accuracy. Most notably, HAG Net performs the best among all the comparison networks, especially in complex areas and high semantic categories (such as roads and buildings), showing excellent detail capture and boundary processing capabilities. The hierarchical semantic segmentation results of HAG remain consistent and highly accurate in multi-category scenarios, validating its advantage as an efficient and stable semantic segmentation method.

[0166] According to the experimental results and network structure analysis in Tables 5 and 6, the HAG network shows outstanding advantages in the water, traffic and farmland categories. HAG achieved an F1 score of 74.60 and an IoU of 59.49 in the water category, which is 31.26 higher than the worst CMLF network in terms of F1 score and 31.83 higher than the worst CMLF network in terms of IoU. In the traffic category, HAG achieved an F1 score of 81.77 and an IoU of 69.16, which is 38.43 higher than the worst CMLF in terms of F1 score and 41.50 higher than the worst CMLF in terms of IoU. In the farmland category, HAG achieved an F1 score of 81.94 and an IoU of 69.41, which is 59.73 higher than the worst CGCL in terms of F1 score and IoU of 56.92. Compared with the best performing networks such as SFFNet, HAG has an IoU of 21.17 higher in the water category, but is slightly lower by 0.40 in the F1 score of the traffic land category. In addition, HAG performs close to the best network in the F1 score of the cultivated land category, with a slight improvement in IoU.

[0167] According to the results in the table and the performance of each network, the HAG network shows excellent segmentation capabilities in multiple categories, especially in the photovoltaic, road and water categories. HAG achieved an F1 score of 84.00 and an IoU of 73.19 in the photovoltaic category, which is 15.17 higher than the worst performing CMLF network in terms of F1 score and 5.91 higher than the worst performing CMLF network. In the road category, HAG's F1 score reached 79.60 and IoU was 61.83, which is 18.90 higher than the worst CGGL network in terms of F1 score and 15.90 higher than the worst CGGL network in terms of IoU. In the water category, HAG's F1 score was 77.88 and IoU was 66.64, which is 56.86 higher than the CMLF network in terms of F1 score and IoU respectively. Compared to the best performing networks such as SFFNet, HAG has a slightly higher IoU of 0.16 in the photovoltaic category, but a lower F1 score of 6.64 in the water category. In the road category, HAG's performance is close to SFFNet, with a higher F1 score of 0.61. HAG's outstanding performance in categories such as photovoltaic, road, and water demonstrates its excellent ability to handle details and capture contextual information in complex scenes. Figure 4It can be seen that different networks show obvious differences in the semantic segmentation task of complex scenes. Deeplabv3+ has a good segmentation effect in homogeneous areas, but there are many misclassifications and discontinuities in complex boundaries and detail areas. CGGL is not good at processing fine boundaries in shengtengcups, and the figure shows that it is slightly weak in capturing subtle structures. The MAE-BG network is more prominent in edge detection and detail retention. The segmentation results are relatively complete when processing areas with complex edges, but there is still room for improvement for very small features. SUNet uses the characteristics of Swin Transformer to perform stably in context modeling, but it is prone to segmentation discontinuities and classification errors in areas with rich details. CMLF performs well in large-area segmentation, but it is slightly insufficient in processing complex boundaries. It can be seen in the figure that its performance in the details is slightly weak. SFFNet has a good performance in utilizing spatial information and processing complex areas, and the edge processing is relatively clear, but the segmentation accuracy in extremely complex scenes is not as good as HAGNet. HAG Net shows excellent segmentation performance, especially when processing highly complex areas, with clear segmentation edges, complete details, and effective reduction of classification errors. It outperforms other networks in multi-category distinction and detail capture, verifying its stability and accuracy as an efficient semantic segmentation method in complex scenes.

[0168] In the ablation experiment section, we will present two aspects in detail: the ablation experiment of the cobweb loss function, and the ablation experiment analysis of each module in the HAG network in the GID dataset. First, we explore the impact of the cobweb loss (L_cobweb) on the overall model performance. The design of L_cobweb aims to enhance the collaboration and interaction modeling capabilities between categories, so as to more accurately capture the complex structure and semantic relationship of the objects. By removing this loss term and conducting experimental comparisons, we can intuitively observe the contribution of L_cobweb to the segmentation results. This study selected the classic Deeplabv3+ network and the MAE-BG network based on the edge algorithm, and performed performance evaluation after adding L_cobweb. The results are shown below. Figure 5 After adding L_cobweb, all evaluation indicators of the two models are improved.

[0169] In the GID-5 test set, different networks showed significant improvements after using L_cobweb. After adding L_cobweb, Deeplabv3+ improved its overall accuracy (OA) by 1.85% to 94.04%; mIoU increased by 3.08% to 90.30%; and FWIoU increased by 3.08% to 88.86%. In the category analysis, the highest improvement was "water", whose OA increased by 4.87%, while mIoU and FWIoU increased by 6.77% respectively. In the MAE-BG network, L_cobweb increased OA from 93.70% to 95.48%, an increase of 1.78%; mIoU increased by 0.65% to 90.41%; and FWIoU increased by 3.02% to 91.38%. Among them, "water" had the highest improvement in OA, mIoU and FWIoU, increasing by 6.92% and 7.82% respectively. In the HAG network, after using L_cobweb, OA increased by 1.26% to 96.74%, mIoU increased by 2.87% to 93.86%, and FWIoU increased by 2.39% to 93.73%. In the category analysis, the highest OA improvement was "built-up", which increased by 2.16%; "meadow" had the highest improvement in mIoU and FWIoU, increasing by 5.73% respectively. In the GID-15 category test set, L_cobweb also significantly improved the performance of each network. In Deeplabv3+, OA increased by 4.12% to 82.24%; mIoU, although the increase was smaller, still increased by 0.39% to 48.51%; FWIoU increased by 2.31% to 70.13%. Among them, the category with the highest improvement was "Urb.land", with an OA increase of 5.07%, while "Pond" had the largest improvement in mIoU and FWIoU, increasing by 1.91%. In the MAE-BG network, after using L_cobweb, OA increased by 3.75% to 83.39%, mIoU increased by 2.18% to 54.34%, and FWIoU increased by 4.68% to 75.17%. The category with the highest improvement was "P.field", whose OA increased by 9.27%; "Garden" had the highest improvement in both mIoU and FWIoU, with an increase of 3.75% respectively. For the HAG network, after using L_cobweb, OA increased by 3.43% to 86.94%, mIoU increased by 4.20% to 58.92%, and FWIoU increased by 4.47% to 79.31%. Category analysis shows that "Urb.land" has the highest improvement in OA, mIoU, and FWIoU, with increases of 11.48% and 9.97% respectively.These results verify the effectiveness of the L_cobweb loss function in different networks, which not only significantly improves the classification performance, but also enhances the segmentation ability and overall performance of the model in complex categories and large-scale scenarios.

[0170] In the Shengteng Cup-8 test set, the L_"cobweb" loss function brought significant improvements in different networks. For the Deeplabv3+ network, after using L_"cobweb", the overall accuracy (OA) increased by 3.40% to 74.52%; mIoU increased by 3.16% to 56.89%; and FWIoU increased by 4.17% to 59.69%. In the category analysis, the category with the highest improvement was "Water", whose OA, mIoU and FWIoU increased by 3.98% and 6.06% respectively. In the MAE-BG network, after introducing L_"cobweb", OA increased by 1.47% to 75.42%; mIoU increased by 4.18% to 56.65%; and FWIoU increased by 1.73% to 60.85%. The category with the highest improvement is "Traffic", whose OA improves by 3.43%, while mIoU and FWIoU improve by 1.64%. For the HAG network, after using L_"cobweb", OA improves by 0.84% ​​to 75.42%; mIoU increases by 1.00% to 56.65%; and FWIoU improves by 1.22% to 60.85%. Among them, the category with the highest improvement is "Bare Soil", whose OA increases by 4.75%, and the category with the largest mIoU improvement is "Traffic", which increases by 4.67%. In the Shengteng Cup-17 test set, the application of the L_"cobweb" loss function in different networks also brings significant improvements. For Deeplabv3+, OA improves by 3.89% to 70.88%; mIoU increases by 3.83% to 48.08%; and FWIoU improves by 4.67% to 55.14%. The category with the highest improvement is "Photovoltaic", whose OA and mIoU increased by 6.85% and 9.44% respectively. In the MAE-BG network, after using L_"cobweb", OA increased by 1.64% to 71.73%; mIoU increased by 2.96% to 51.85%; FWIoU increased by 1.75% to 56.14%. In this network, the category with the highest improvement is "Parking lot", whose OA increased by 7.01% and mIoU increased by 3.21%. For the HAG network, the L_"cobweb" loss function increased OA by 3.12% to 73.67%; mIoU increased by 3.47% to 54.58%; FWIoU increased by 3.87% to 58.60%. Among them, the "Photovoltaic" category has the highest improvement, with OA increased by 3.30% and mIoU increased by 1.47%.These results verify the effectiveness of the L_"cobweb" loss function in improving the classification capabilities of different networks and the overall segmentation performance.

[0171] In the experiments on the GID dataset, we conducted ablative experiments on the key modules introduced by the HAG network one by one to analyze the impact of these modules on the overall network performance. Specifically, this part of the experiment includes removing or replacing the multi-scale feature extraction module, attention mechanism module, and category relationship modeling module in the HAG network, and observing their impact on the accuracy of object segmentation, IoU, and F1 score. By gradually removing these modules, we can clarify the role of each module in capturing contextual information, enhancing semantic consistency, and improving the performance of complex region segmentation. The baseline model is ResUNet, the ED module combines the Residual block and the ECA module, MRB is a multi-scale Residual block, GA-Block is a gated attention mechanism module, and Lcobweb is a dynamic spider web loss module. In the GID-5 test set, with the gradual introduction of modules, the performance of the HAG network gradually improves. After adding the ED module, the overall accuracy (OA) is increased to 95.48%, and the mIoU and FWIoU also increase to 90.99% and 91.34% respectively. After the introduction of the MRB module, OA further increased to 96.14%, and mIoU and FWIoU increased to 92.85% and 92.59% respectively. When GA-Block was added, although OA dropped slightly to 96.01%, the overall performance remained stable, and GA-Block was proved to be more suitable for multi-category classification. Finally, when the Cobweb module was introduced into the full configuration, OA increased to 96.74%, and mIoU and FWIoU reached 93.86% and 93.73% respectively. In the GID-15 class test set, the gradual addition of different modules also brought significant improvements. After adding the ED module, OA increased to 83.52%, and mIoU and FWIoU increased to 54.72% and 74.84% respectively. After the introduction of the MRB module, OA increased to 84.24%, and mIoU and FWIoU reached 55.53% and 76.34% respectively. The addition of GA-Block further improves OA to 85.07% and FWIoU slightly improves to 76.69%. After adding the Cobweb module to the full configuration, OA is significantly improved to 86.94%, mIoU reaches 58.92%, and FWIoU rises to 79.31%. Overall, in the GID-5 test set, the fully configured HAG network improves OA by 1.49%, mIoU by 2.89%, and FWIoU by 2.68% compared to the Baseline; in the GID-15 test set, the fully configured HAG network improves OA by 4.47%, mIoU by 4.91%, and FWIoU by 5.64%.These results show that the gradual introduction of ED, MRB, GA-Block and Cobweb modules can significantly improve the classification performance of the HAG network on the GID dataset, especially in the full configuration, where the network exhibits the highest accuracy and consistency.

[0172] Conclusion: By introducing hierarchical dependencies between categories and a dynamic loss adjustment mechanism, the deep semantic segmentation method proposed in this paper shows significant advantages in the high-resolution remote sensing image segmentation task. This method uses a special category transformation matrix to capture the specific relationship between categories and dynamically integrates them into the loss function to achieve mandatory constraints and collaborative optimization between multiple categories. This makes the segmentation process not only efficient in capturing contextual information, but also shows extremely high accuracy and stability in complex backgrounds and multi-level category relationship scenes.

[0173] Experimental results show that our method exhibits excellent performance in both coarse-grained and fine-grained category segmentation tasks, demonstrating its outstanding ability in fine classification and complex scene processing. In particular, by introducing a dynamic loss adjustment mechanism, our method reaches a new level in category relationship modeling and multi-scale feature fusion, effectively reducing segmentation errors and enhancing the clarity of edge processing. This innovative interactive optimization process allows the relationship between ancestor and subcategories to influence each other during the segmentation process, thereby improving the overall segmentation performance.

[0174] In addition, the experimental results further verify the key role of each module in improving the segmentation performance and demonstrate the unique contribution of the cobweb loss in capturing the structure of ground objects and the interaction between categories. Future work will focus on exploring more complex category hierarchies and optimizing dynamic loss adjustment strategies to further improve segmentation accuracy and the generalization ability of the model. We also plan to extend this method to more types of remote sensing image data to verify its applicability in different environments and application scenarios. Through continuous improvement, we expect to continue to achieve higher classification accuracy and model stability when dealing with more complex and diverse remote sensing tasks.

[0175] Compared with the classic model:

[0176] The visualization of our network in remote sensing image segmentation, especially under the influence of our proposed Cobweb loss, shows several significant improvements. First, for the farmland part, traditional segmentation methods are often challenged by the different spectral color changes caused by seasonal changes, which makes it difficult to accurately identify the farmland area. However, with the introduction of Cobwebloss, our network successfully overcomes this challenge and achieves accurate segmentation of the farmland area. Secondly, after being processed by our network, the water area shows clearer edge positioning, significantly reducing the misclassification problem between buildings and other categories such as water bodies and grasslands. This is because Cobweb loss can effectively understand and utilize the hierarchical relationship between categories and boundary constraints, thereby improving the model's ability to handle edge details.

[0177] In terms of fine-category segmentation, especially road areas, traditional annotation usually labels surrounding areas as roads to simplify operations, resulting in blurred boundaries during segmentation. Our network can not only accurately divide road areas through the interaction of Cobweb loss and six-category information, but also effectively distinguish adjacent non-road areas, significantly improving the classification accuracy of Irrigate land and Traffic land. This advantage stems from our network's ability to model hierarchical dependencies between categories, which can better handle complex land types and provide more accurate semantic segmentation results.

[0178] The effect of adding Cobweb loss to the classic network deeplabv3+ is slightly improved. Compared with the results of adding cobweb loss to other networks, it is found that the basic classification confusion caused by the hollow convolution in the hollow space pyramid pooling of deeplabv3+ makes the constraint of cobweb loss not achieve good results. Secondly, the regularity is improved. For the entire data set, some categories are reduced and do not conform to the overall regularity of some samples in the test set. By introducing the hierarchical dependency relationship between categories and the dynamic loss adjustment mechanism, the deep semantic segmentation method proposed in this paper shows excellent segmentation performance on high-resolution remote sensing images, especially when dealing with scenes with complex backgrounds and multi-level category relationships. Future work will explore more complex category hierarchies and loss adjustment strategies to further improve segmentation accuracy and the generalization ability of the model.

[0179] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0180] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

[0181] Table 1. Evaluation of GID-5 test set on OA, Recall

[0182]

[0183] Table 2. Evaluation of the GID-5 test set on F1, MIOU, and FWIOU

[0184]

[0185] Table 3. Evaluation of the accuracy of the GID-15 test set

[0186]

[0187]

[0188] Table 4. Evaluation of GID-15 dataset on IOU

[0189]

[0190] Table 5. Evaluation of the Sheng teng Cup-8 test set on precision and recall

[0191]

[0192]

[0193] Table 6. Evaluation results of Sheng teng Cup-8 test set on F1, IOU and FWIOU

[0194]

[0195] Table 7. Evaluation results of Sheng teng Cup-17 test set on accuracy and F1 value

[0196]

[0197]

[0198] Table 8. Evaluation results of Sheng teng Cup-17 test set on accuracy, IOU and FWIOU values

[0199]

[0200] Table 9. Evaluation of HAG network on GID test set (coarse categories)

[0201]

[0202] Table 10. Evaluation of HAG network on GID test set (fine categories)

[0203]

[0204] The above Tables 1 to 10 include two data sets, namely: the GID test set and the Sheng teng Cup test set.

[0205] The 6 Chinese labels in the GID-5 test set are: Building (built-up), Farmland, Forest, Meadow, and Water. The 15 Chinese labels in the GID-15 test set are: Industrial land, Urban residential, Rural residential, Traffic land, Paddy field, Irrigated land, Dry cropland, Garden plot, Arborwoodland, Shrub land, Natural grassland, Artificial grassland, River, Lake, and Pond.

[0206] Similarly, the 7 Chinese labels in the Sheng teng Cup-8 test set are: Water, Traffic, Farmland, Grassland, Woodland, Bare Soil, Others. The 17 Chinese labels in the Sheng teng Cup-17 test set are: Water, Road, Airport, Train Station, Building, Solar Panel, Parking Lot, Playground, Farmland, Greenhouse, Natural Grassland, Lawn, Natural Forest, Artificial Forest, Natural Bare Soil, Artificial Bare Soil, Others.

Claims

1. A deep hierarchical multi-semantic data processing method based on the collaboration of dynamic spider web and loss function, characterized in that: This deep hierarchical multi-semantic data processing method is applicable to the hierarchical semantic segmentation network HAGNet model built on the ResUNet network with an encoder-decoder structure. Its encoder part includes a multi-scale residual block and an edge enhancement module, its bridge part uses a common convolutional network and a dilated spatial pyramid pooling ASPP module, and its decoder part consists of a multi-scale residual block and an attention gate. The deep layered multi-semantic data processing method includes: In the encoder part, multi-scale residual blocks are used to capture feature information of different scales, and a spider web structure is used to represent the potential hierarchical dependencies between semantic categories. The associated weights are formalized through matrices. The edge enhancement module is used to enhance the edge information of the details and improve the recognition ability of the target boundary. In the bridge part, the different-sized atrous convolution kernels in the ASPP module are used to capture global context information of different scales and improve the global feature capability of the model. In the decoder part, a multi-scale residual block is used to restore the spatial resolution of the feature map layer by layer to extract feature information, and then concatenate it with the edge features from the encoder through an attention gate; The dynamically adjusted optimization loss function is defined to coordinate the cobweb loss. The optimization loss function is constructed based on the basic loss function and pre-calculated association weights. The association weight matrix reflects the hierarchical dependency between semantic categories and is used to balance the weights of different semantic categories in the segmentation process.

2. The method for processing deeply layered multi-semantic data according to claim 1, characterized in that: The matrix is ​​denoted as M cf , the associated weight is recorded as W cf , through the matrix W cf To formalize the association weight W cf ; Where each element of the matrix W cf represents the association weight between the coarse category c and the detailed category f; Association weight W cf The calculation is based on pixel matching statistics: W cf =M cf [c, f], where M cf [c, f] represents the number of pixel matches between category c and category f, which is accumulated by traversing all images and their corresponding labels.

3. The method for processing deeply layered multi-semantic data according to claim 2, characterized in that: The optimization loss function is denoted as L cobweb , the basic loss function is recorded as L base , the dynamic factor is recorded as δ, and the optimization loss function is expressed as: L cobweb =L base +δ×(1-R), where the relationship ratio R is a quantitative value that quantifies the degree of dependence between semantic categories; In the image semantic segmentation task, the optimal segmentation result H*(I) is expressed as the optimal label set for each pixel i in image I. in By optimizing the loss function L cobweb The optimal label obtained to minimize the loss function; The optimization process of the image semantic segmentation task is expressed as:

4. The method for processing deeply layered multi-semantic data according to claim 3, characterized in that: When predicting from a subdivided category f to a coarse category c, the relationship ratio R of category c to category f cf The calculation method is as follows: W cf =M cf [c,f], where M c，f is an element in the mapping matrix from subdivided category f to coarse category c, indicating the mapping strength of category f to category c. The denominator is for a given coarse category C, where F represents the total number of subdivided categories, ensuring that the relationship ratio is normalized; At this point, the basic loss function and the adjusted fine category loss function are: and Where N is the total number of pixels in the image, is the model predicting that the nth pixel belongs to its true category y n The probability of y n is the true category index of the pixel, belonging to the subdivided category set, and ∈ is a very small constant used to ensure numerical stability.

5. The method for processing deeply layered multi-semantic data according to claim 3, characterized in that: When predicting from coarse category c to detailed category f, the relationship ratio R of category f to category c fc The calculation method is as follows: W cf =M cf [c,f]; where M c，f is an element in the mapping matrix from subdivided category f to coarse category c, indicating the mapping strength of subdivided category f to coarse category c, and the denominator is for a given subdivided category f, where c represents the total number of coarse categories; At this point, the basic loss function and the adjusted fine category loss function are: and, Where N is the total number of pixels in the image, is the model predicting that the nth pixel belongs to its true category y n The probability of y n is the true category index of the pixel, belonging to the coarse classification set, ∈ is a very small constant used to ensure numerical stability.

6. The method for processing deeply layered multi-semantic data according to claims 2, 4 and 5, characterized in that: There are 6 coarse categories and 16 detailed categories.

7. The method for processing deeply layered multi-semantic data according to claim 1, characterized in that: In the encoder part, the multi-scale residual block includes three parallel convolution paths, 3x3, 5x5 and 7x7 convolutions, which capture feature information of different scales through multi-resolution paths and enhance the overall performance.

8. The method for processing deeply layered multi-semantic data according to claim 1, characterized in that: The number of edge enhancement modules is 4; the edge enhancement module is obtained by embedding a lightweight attention ECA module after the ordinary residual block and the edge extraction convolution block; In the decoding part, a multi-scale residual block is used to restore the spatial resolution of the feature map layer by layer, which goes through five decoding steps in total; In the decoding process of each layer, the current feature map is concatenated with the corresponding skip connection feature map and enhanced by the attention gate to highlight the important feature areas; Each layer of the decoder will also concatenate the feature map of the current layer with the edge features from the encoder and the feature information of the multi-scale residual block MultiRes Block, thereby further fusing the multi-scale feature information.

Images