Dynamic spider-web and loss function collaboration deep hierarchical multi-semantic data processing method

Abstract

The application discloses a kind of dynamic spider webs and loss function cooperation's deep hierarchical multi-semantic data processing method, can effectively restore the spatial resolution and detail information of image, improve the precision and robustness of segmentation result.The method comprises: different scale feature information is captured by multi-scale residual block, and potential hierarchical dependency between semantic categories is represented using spider web structure, and the correlation weight is formalized by matrix;Edge information in detail part is lifted and strengthened by edge enhancement module, and the identification ability to target boundary is improved;Different size hole convolution kernel in ASPP module is used to capture global context information of different scales, and the global feature capability of the model is improved;Multi-scale residual block is used to recover spatial resolution of feature map layer by layer to extract feature information, and edge features from the encoder are spliced through attention gate;Dynamic adjusted optimization loss function is defined to coordinate and adjust the spider web loss.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of remote sensing image semantic segmentation, and particularly relates to a deep hierarchical multi-semantic data processing method based on dynamic spider web and loss function cooperation. BACKGROUND

[0002] Semantic segmentation aims to assign semantic labels to pixels, which is crucial in remote sensing image interpretation. Most segmentation deep neural networks assume no interaction and synergy among target classes in design, leading to cumulative segmentation errors and failing to capture the essence of the structure of the visual world.

[0003] Existing semantic segmentation research has limitations in integrating semantic hierarchy, such as methods for natural scenes not being applicable to remote sensing images, or complex framework modifications, and most of them do not fully consider label dependency and interaction. In the field of remote sensing, capturing and representing structured information is crucial for adapting to new environments and multi-level abstract understanding.

[0004] Remote sensing images have complex data properties, such as heterogeneity, class imbalance, and overlapping class conditional distribution, and imaging conditions can lead to photographic distortion, scale variation, and illumination variation, reducing class separability. Traditional hierarchical semantic segmentation methods face challenges in remote sensing images, with irreversible inter-class membership, and relying solely on single semantic hierarchy division can lead to inaccurate segmentation. SUMMARY

[0005] To overcome the above technical defects, a deep hierarchical multi-semantic data processing method based on dynamic spider web and loss function cooperation is provided in the embodiment of the present application. The specific scheme is as follows:

[0006] The deep hierarchical multi-semantic data processing method is applicable to a hierarchical semantic segmentation network HAG Net model constructed based on a ResUNet network with a codec structure. The encoder part includes multi-scale residual blocks and edge enhancement modules, the bridge part uses a normal convolution network and an ASPP module, and the decoder part is composed of multi-scale residual blocks and attention gates.

[0007] The deep hierarchical multi-semantic data processing method comprises:

[0008] In the encoder part, multi-scale residual blocks are used to capture feature information at different scales, and a spider web structure is used to represent the potential hierarchical dependency between semantic classes, with matrix-form correlation weights. The edge enhancement module is used to enhance the edge information in the detail part and improve the recognition ability of the target boundary.

[0009] In the bridge part, different size of the kernel in the ASPP module is used to capture global context information at different scales, and the global feature ability of the model is improved.

[0010] In the decoder part, the multi-scale residual block is used to recover the spatial resolution of the feature map layer by layer to extract feature information, and the attention gate is used to splice the edge features from the encoder;

[0011] The optimization loss function is used to cooperatively adjust the cobweb loss, and the optimization loss function is constructed based on the basic loss function and the pre-calculated correlation weight, and the correlation weight matrix reflects the hierarchical dependency relationship between semantic categories, and is used to balance the weight of different semantic categories in the segmentation process.

[0012] Optionally, in a possible implementation, the matrix is denoted as M cf , the correlation weight is denoted as W cf , and the correlation weight W cf is formalized by the matrix M cf ;

[0013] Wherein each element W cf of the matrix represents the correlation weight between the coarse category c and the fine category f.

[0014] The calculation of the correlation weight W cf is based on pixel matching statistics: W cf =M cf [c,f], wherein M cf [c,f] represents the number of pixel point matches between the category c and the category f, which is accumulated by traversing all images and their corresponding labels.

[0015] Optionally, in a possible implementation, the optimization loss function is denoted as L cobweb , the basic loss function is denoted as L base , the dynamic factor is denoted as δ, and the optimization loss function is represented as: L cobweb =L base +δ×(1-R), wherein the relationship ratio R is a quantitative value quantifying the degree of dependency between semantic categories.

[0016] In the image semantic segmentation task, the optimal segmentation result H*(I) is represented as the optimal label set of each pixel i in the image I Wherein is the optimal label obtained by minimizing the loss function L cobweb ;

[0017] The optimization process of the image semantic segmentation task is represented as:

[0018]

[0019] Optionally, in a possible implementation, when predicting from the fine category f to the coarse category c, the relationship ratio R of the category c to the category fcf The calculation method is as follows: W cf = M cf [c,f], wherein M c，f is an element in the mapping matrix from the fine category f to the coarse category c, representing the mapping strength of the category f to the category c, and the denominator is the total number of fine categories F for a given coarse category c, which ensures that the relationship ratio is normalized;

[0020] At this time, the basic loss function and the adjusted fine category loss function are respectively:

[0021] and wherein N is the total number of pixels in the image, is the probability that the model predicts that the nth pixel belongs to its true category y n , y n is the true category index of the pixel, and belongs to the set of fine categories, and ∈ is a very small constant to ensure numerical stability.

[0022] Optionally, in a possible implementation, when predicting from the coarse category c to the fine category f, the relationship ratio R fc of the category f to the category c is calculated as follows:

[0023] W cf = M cf [c,f]; wherein M c，f is an element in the mapping matrix from the fine category f to the coarse category c, representing the mapping strength of the fine category f to the coarse category c, and the denominator is the total number of coarse categories c for a given fine category f;

[0024] At this time, the basic loss function and the adjusted fine category loss function are respectively:

[0025] and wherein N is the total number of pixels in the image, is the probability that the model predicts that the nth pixel belongs to its true category y n , y n is the true category index of the pixel, and belongs to the set of coarse categories, and ∈ is a very small constant to ensure numerical stability.

[0026] Optionally, in a possible implementation, the coarse category is 6, and the fine category is 16.

[0027] Optionally, in a possible implementation, in the encoder part, the multi-scale residual block includes three parallel convolution paths, 3x3, 5x5 and 7x7 convolutions, different scale feature information is captured through the multi-resolution path, and the overall performance is enhanced.

[0028] Optionally, in a possible implementation, the number of edge enhancement modules is 4; the edge enhancement module is obtained by embedding a lightweight attention ECA module after a normal residual block and an edge extraction convolution block;

[0029] In the decoding part, the feature map is recovered layer by layer through five decoding steps in total by using the multi-scale residual block;

[0030] In each layer of the decoding process, the current feature map is spliced with the corresponding skip connection feature map, and is enhanced through the attention gate, so as to highlight the important feature area;

[0031] Each layer of the decoder also splices the feature map of the current layer with the edge feature from the encoder and the feature information of the multi-scale residual block MultiRes Block, so as to further fuse the multi-scale feature information.

[0032] The technical solutions adopted in the embodiments of the application can achieve the following technical effects:

[0033] The boundary enhancement module aims to strengthen the edge information of feature extraction, so as to improve the recognition ability of the model to the target boundary;

[0034] The multi-scale residual block captures multi-level features in the image through convolution operations of different scales, and enhances the expression ability of the features.

[0035] The ASPP module can capture different scale global context information through different size of hole convolution kernels, and further improves the global feature extraction ability of the model.

[0036] The decoder uniformly uses the MultiRes Block to recover the spatial resolution layer by layer, and totally undergoes five decoding steps. In each layer of the decoding process, the current feature map is spliced with the corresponding skip connection feature map, and is enhanced through the attention gate (AG) to highlight the important feature area. Each layer of the decoder also splices the feature map of the current layer with the edge feature from the encoder and the feature information of the MultiRes Block, so as to further fuse the multi-scale feature information. This structure design ensures that under the condition of complex and variable input data, the decoder can effectively recover the spatial resolution and detail information of the image, and improve the precision and robustness of the segmentation result.

[0037] By adopting a non-tree structure (such as a cobweb structure) to represent the relationship between categories and utilizing a category transition matrix to formalize this relationship, where the matrix elements represent the association weight between a coarse category c and a fine category f, the association weight is calculated based on pixel matching statistics. This way can better capture the hierarchical dependency relationship between categories, and find the most suitable label for each pixel in the image under the framework of hierarchical dependency and dynamic loss adjustment.

[0038] Adjusting the loss according to the category relationship: by calculating the relationship ratio to quantify the degree of dependency, when predicting from fine categories to coarse categories, the adjusted fine category loss function is obtained by adjusting the relationship ratio; when predicting from coarse categories to fine categories, the adjusted coarse category loss function is obtained by adjusting the relationship ratio. This way of dynamically adjusting the loss function according to the category relationship can better adapt to the complex relationship between different categories and improve the performance of the model in multi-category segmentation tasks. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are merely examples and do not limit the scope of the claims. In all the drawings, like reference numerals refer to like but not necessarily identical elements.

[0040] Figure 1 A flowchart of the deep hierarchical multi-semantic data processing method of dynamic cobweb and loss function cooperation in this application;

[0041] Figure 2 A network structure diagram of the HAG Net model in this application;

[0042] Figure 3 For the present application A diagram showing the performance of traditional and advanced networks on the GID test set;

[0043] Figure 4 A diagram showing the performance of traditional and advanced networks on the shengtengcup test set;

[0044] Figure 5 A diagram showing the performance of traditional and advanced networks on the Gid and shengtengcup test sets before and after adding Cobweb loss. DETAILED DESCRIPTION

[0045] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0046] It should be noted that the description of "first", "second" and the like in the embodiments of the present application is only for the purpose of description and cannot be understood as indicating or implying the relative importance of the technical features indicated or implying the number of technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that the technical solutions can be realized by those of ordinary skill in the art. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection claimed by the present application.

[0047] In the description of the present application, it should be understood that the numerical reference before the step does not indicate the order before and after the execution of the step, but is only used to facilitate the description of the present application and to distinguish each step, and therefore cannot be understood as a limitation of the present application.

[0048] Deep learning-based methods are widely used in semantic segmentation, but only by increasing the number of layers to deepen the network will face challenges, such as deep network relying on a small local receptive field, limiting the ability to understand the global context. Scholars have designed various solutions, such as expanding the receptive field, building a pyramid structure, using edge perception or boundary constraints, using encoder-decoder architecture, introducing attention mechanism, and using incremental learning. In the field of remote sensing image processing, traditional segmentation methods assign each pixel to a single class label, which cannot effectively express complex relationships and weaken the understanding of the structural integrity of the ground object.

[0049] Convolutional networks can automatically learn hierarchical feature representations, and common traditional methods such as using Markov random fields and conditional random fields and designing cascaded classifiers have limited ability to obtain context information at multiple scales. Multi-scale processing is important to preserve the context information of the image, and multiple structures can be used to integrate multi-scale context information. With the improvement of satellite image resolution, semantic-level analysis algorithms have gradually developed, but there is a semantic gap. Common coarse-to-fine level methods include recursive coarse-to-fine framework based on image-level class labels, introducing Bayesian techniques into hierarchical classification, knowledge extraction constraints, etc. In addition, segmentation can also be performed through methods such as saliency map and hierarchical region tree, but these methods have problems such as sensitivity to parameter selection, dependence on the accuracy of initial labels, and high computational complexity.

[0050] For the hierarchical semantic segmentation problem mentioned above, a new hierarchical semantic segmentation network (Hierarchical Attention Gate-MultiResUNet, HAG Net) model is proposed in the embodiments of the present application, which utilizes structured class relationships and dynamic factor adjustment loss function to improve model performance. Unlike previous structured segmentation models that focus on complex network design, the hierarchical semantic segmentation method is formalized as a pixel-based multi-label classification task, the relationship is inferred to reflect the potential relationship of the hierarchy, and the segmentation is forced to comply with the hierarchy structure, and the class hierarchy knowledge is encoded into the pixel embedding space.

[0051] First, in order to facilitate the understanding of the technical solutions in the embodiments of the present application, the technical terms involved in the present solution will be explained first:

[0052] 1. ResUNet network:

[0053] ResUNet is a deep learning model that combines ResNet and U-Net structure, commonly used for image segmentation tasks. It integrates the encoder-decoder structure of U-Net and the residual connection characteristics of ResNet, enabling it to effectively propagate gradients in deep networks, alleviating the problem of gradient vanishing, and improving the stability and performance of the model.

[0054] 2. Atrous Spatial Pyramid Pooling (ASPP) module:

[0055] ASPP module is a technique for expanding the receptive field of convolution, which can effectively capture multi-scale context information. By using different sampling rates of atrous convolution to process input feature maps in parallel, ASPP can enhance the model's ability to detect targets of different scales in images, especially suitable for image segmentation tasks in complex scenes.

[0056] 3. MultiRes Block:

[0057] MultiRes Block is an improved residual module designed to capture features at multiple resolutions simultaneously. By combining convolution kernels of different sizes within a single module, MultiRes Block can improve the diversity and accuracy of feature extraction while maintaining residual connections, making the model more advantageous when dealing with images rich in details.

[0058] 4. skip connection feature map:

[0059] Skip connection is a structure widely used in U-Net and other networks to pass feature maps between encoders and decoders. It helps preserve original input features by directly connecting inputs and outputs between different network layers, enhancing the decoder stage's ability to recover contextual information and improving the accuracy of segmentation results.

[0060] Secondly, in order to facilitate those skilled in the art to understand the technical solutions in the embodiments of the present application, the HAG Net model and the deep hierarchical multi-semantics data processing method thereof proposed in the present application are described as follows:

[0061] Embodiment one

[0062] As shown in Figure 1 The deep hierarchical multi-semantics data processing method in the embodiments of the present application is applicable to the hierarchical semantic segmentation network HAG Net model constructed based on the ResUNet network with encoder-decoder structure. The encoder part includes multi-scale residual blocks and edge enhancement modules. The bridge part uses a normal convolution network and an Atrous Spatial Pyramid Pooling (ASPP) module. The decoder part is composed of multi-scale residual blocks and attention gates.

[0063] The deep hierarchical multi-semantics data processing method includes:

[0064] S1, in the encoder part, multi-scale residual blocks are used to capture feature information of different scales, and a spider web structure is used to represent the potential hierarchical dependency relationship between semantic categories, and the correlation weight is formalized in matrix form. The edge information in the detail part is extracted and strengthened by the edge enhancement module, and the recognition ability of the target boundary is improved.

[0065] Compared with the ordinary residual block, the MultiRes Block captures feature information of different scales through multi-resolution paths, enhancing the feature extraction capability. The multi-scale residual block is applied to the main network architecture to improve the overall performance of the model by capturing multi-scale features.

[0066] The edge enhancement module uses an ordinary residual block to improve the network's ability to capture edge information. Optionally, the number of edge enhancement modules is 4. The edge enhancement module is obtained by embedding an Efficient Channel Attention (ECA) module after the ordinary residual block and the edge extraction convolution block.

[0067] Optionally, in some embodiments of the present application, in the encoder part, the multi-scale residual block contains three parallel convolution paths: the first path is a 3x3 convolution, the second path is a 5x5 convolution, and the third path is a 7x7 convolution, where the filter allocation is int(w*0.167), int(w*0.333), and int(w*0.5), respectively. Through this design, the network can capture feature information of different scales at the same time, enhancing the understanding of image details.

[0068] Specifically, the matrix is denoted as M cf , the association weight is denoted as W cf , and the association weight W cf is formalized by the matrix M cf , where each element W cf of the matrix represents the association weight between the coarse category c and the fine category f; the calculation of the association weight W cf is based on pixel matching statistics: W cf = M cf [c, f], where M cf [c, f] represents the number of pixel point matches between the category c and the category f, which is accumulated by traversing all images and their corresponding labels.

[0069] Optionally, in some embodiments, the coarse category is 6, and the fine category is 16.

[0070] S2, in the bridge part, different size of the hole convolution kernel in the ASPP module is used to capture the global context information of different scales, and the global feature ability of the model is improved.

[0071] The bridge part adopts a common convolution network and an ASPP module. The ASPP module can capture global context information of different scales through different size of the hole convolution kernel, further improving the global feature extraction ability of the model.

[0072] S3, in the decoder part, a multi-scale residual block is used to recover the spatial resolution of the feature map layer by layer to extract feature information, and the edge features from the encoder are spliced through the attention gate.

[0073] The decoder uniformly adopts the MultiRes Block to recover the spatial resolution layer by layer, and a total of five decoding steps are performed. In each layer of the decoding process, the current feature map is spliced with the corresponding skip connection feature map, and is enhanced through the attention gate (AG) to highlight important feature regions. Each layer of the decoder also splices the current layer feature map with the edge features from the encoder and the MultiRes Block feature information, thereby further fusing multi-scale feature information. This structural design ensures that the decoder can effectively recover the spatial resolution and detail information of the image under complex and variable input data conditions, improving the accuracy and robustness of the segmentation result.

[0074] The two inputs of the attention gate module are skip_connection and gating_signal, and W x and W g The main structure is composed of a convolution 1x1 and an activation function, and finally, the attention weight map is up-sampled (upConv_Block) and de-convolved (trans_conv2D) to ensure that its size matches the skip_connection. By dynamically adjusting the weight of the features through the gating unit, the gated attention mechanism can not only effectively enhance the attention to multiple targets, but also maintain the balance of attention between different categories in a multi-target scene, thereby significantly improving the overall model performance. Through this method of dynamically adjusting the feature weight, the gated attention mechanism can maintain high accuracy and performance when processing complex multi-target tasks.

[0075] S4, use the defined dynamic adjusted optimization loss function to cooperatively adjust the spider web loss.

[0076] The optimization loss function is constructed based on the basic loss function, the pre-calculated correlation weight and the dynamic factor. The dynamic factor can adjust the optimization loss function according to the average loss of multiple cycles adjacent to the current training cycle. The correlation weight matrix reflects the hierarchical dependency relationship between semantic categories, and is used to balance the weights of different semantic categories in the segmentation process.

[0077] Optionally, in some embodiments, the optimization loss function is denoted as L cobweb , the basic loss function is denoted as L base , and the dynamic factor is denoted as δ. The optimization loss function is represented as: L cobweb = L base + δ × (1-R), where the relationship ratio R is a quantitative value quantifying the degree of dependence between semantic categories.

[0078] In the image semantic segmentation task, the optimal segmentation result H*(I) represents the optimal label set for each pixel i in the image I where is obtained by optimizing the loss function L cobweb to minimize the loss function;

[0079] The optimization process of the image semantic segmentation task is represented as:

[0080]

[0081] Further, when predicting from a fine category f to a coarse category c, the relationship ratio R cf of the category f to the category c is calculated as follows: W cf = M cf [c,f], where M c，f is an element in the mapping matrix from the fine category f to the coarse category c, representing the mapping strength of the fine category f to the coarse category c, and the denominator is the total number of fine categories F for a given coarse category c, ensuring that the relationship ratio is normalized;

[0082] At this time, the basic loss function and the adjusted fine category loss function are respectively:

[0083] and where N is the total number of pixels in the image, is the probability that the model predicts the nth pixel to belong to its true category y n , y n is the true category index of the pixel, belonging to the fine category set, and ∈ is a very small constant to ensure numerical stability.

[0084] Further, when predicting from a coarse category c to a fine category f, the relationship ratio R fc of the category f to the category c is calculated as follows:

[0085] W cf = M cf [c,f]; where M c，f is an element in the mapping matrix from the fine category f to the coarse category c, representing the mapping strength of the fine category f to the coarse category c, and the denominator is the total number of coarse categories C for a given fine category f;

[0086] At this time, the basic loss function and the adjusted fine category loss function are respectively:

[0087] and, where N is the total number of pixels in the image, is the model's prediction of the probability that the nth pixel belongs to its true class y n n is the true class index of the pixel, belongs to the set of coarse classes, is a very small constant to ensure numerical stability.

[0088] Finally, the update of the dynamic factor δ is based on the average loss of the past 5 training periods, which is calculated as follows:

[0089] In the technical solution of the present application, the improvement of the optimized loss function cooperatively adjusting the spider web loss is reflected in the following aspects:

[0090] 1. Considering the hierarchical dependency relationship between classes

[0091] Limitations of traditional methods: Traditional segmentation methods often do not fully consider the hierarchical dependency relationship between classes when dealing with semantic segmentation tasks. For example, in single-layer semantic segmentation, each pixel is simply classified into a predefined semantic class without considering the relationship between classes. This approach cannot accurately capture the structural nature of the visual world when dealing with complex remote sensing image data, resulting in cumulative segmentation errors.

[0092] Improvements of dynamic spider web model: By using a non-tree structure (such as a spider web structure) to represent the relationship between classes and using a class transition matrix to formalize this relationship, the matrix elements represent the association weight between coarse class c and fine class f, and the association weight is calculated based on pixel matching statistics. This approach can better capture the hierarchical dependency relationship between classes and find the most suitable label for each pixel in the image under the framework of hierarchical dependency and dynamic loss adjustment.

[0093] 2. Dynamically adjusting the loss function

[0094] Limitations of traditional loss functions:

[0095] Traditional loss functions (such as cross-entropy loss sparse_categorical_crossentropy) only consider the difference between the predicted value and the true value of the pixel itself, which can lead to a decline in model performance when dealing with complex class imbalance and multi-class segmentation problems.

[0096] Adjusting the loss according to the class relationship: By calculating the relationship ratio to quantify the dependency, when predicting from fine classes to coarse classes, the adjusted fine class loss function is obtained by adjusting the relationship ratio; when predicting from coarse classes to fine classes, the adjusted coarse class loss function is obtained by adjusting the relationship ratio. This way of dynamically adjusting the loss function according to the class relationship can better adapt to the complex relationship between different classes and improve the performance of the model in multi-class segmentation tasks.​

[0097] Embodiment Two

[0098] The codec structure is a simple and effective semantic segmentation structure. The encoder part gradually reduces the spatial dimension through the pooling layer, and obtains the features from shallow to deep. The decoder gradually recovers the target details and spatial dimensions through deconvolution or interpolation. It also ensures that the output has the same dimension as the input image, so that end-to-end training can be realized. There is usually a shortcut connection from the encoder to the decoder to help the decoder better recover the object details.

[0099] The simple form of the encoder-decoder structure often loses global information because it directly fuses the deepest features without any other possible operations. This leads to the fact that in dense classification of high-resolution aerial images, the pixels inside large-scale buildings are misclassified.

[0100] Therefore, it is necessary to design a network that can both take advantage of the stability of the codec structure and avoid losing global features, i.e. the HAG Net model in the present application is also a ResUNet network based on the codec structure, and the network structure is as shown in Figure 2

[0101] The hierarchical semantic segmentation network HAG Net proposed in the present application is mainly based on the ResUNet network. The encoder part is composed of 4 boundary enhancement modules and multi-scale residual blocks. The boundary enhancement module aims to strengthen the edge information of feature extraction, so as to improve the recognition ability of the model to the target boundary. The multi-scale residual block captures multi-level features in the image through different scale convolution operations, and enhances the expression ability of the features. The bridge part adopts an ordinary convolution network and an Atrous Spatial Pyramid Pooling (ASPP) module. The ASPP module can capture different scale global context information through different size of the atrous convolution kernel, and further improves the global feature extraction ability of the model. The decoder uniformly adopts the multi-scale residual module MultiRes Block for layer-by-layer recovery of the spatial resolution, and a total of five decoding steps are performed. In each decoding process, the current feature map is spliced with the corresponding skip connection feature map, and is enhanced through the attention gate (AG) to highlight the important feature area. Each layer of the decoder also splices the feature map of the current layer with the edge feature from the encoder and the feature information of the MultiRes Block, so as to further fuse the multi-scale feature information. This structure design ensures that the decoder can effectively recover the spatial resolution and detail information of the image under the condition of complex and variable input data, and improves the precision and robustness of the segmentation result.

[0102] Ordinary residual block and multi-scale residual block:

[0103] ​In Figure 2 Two kinds of residual blocks are mainly used in the HAG Net network structure shown, one is an edge residual block applied to edge enhanced, and the other is a multi-scale residual block applied to the main network architecture. The edge residual block applied to the edge enhanced block is composed of a basic block. Compared with the ordinary residual block, the multi-resolution residual block (MultiRes Block) captures feature information of different scales through a multi-resolution path, enhancing the feature extraction capability. While the ordinary residual block solves the gradient disappearance problem in the deep network through a shortcut connection, but it is not as detailed and diverse as the multi-resolution block in feature extraction. The residual block applied to the edge enhanced block is usually composed of a basic block, which is used to improve the network's ability to capture edge information; while the multi-scale residual block is applied to the main network architecture, which improves the overall performance of the model by capturing multi-scale features. The multi-resolution residual block contains three parallel convolution paths: the first path is a 3x3 convolution, the second path is a 5x5 convolution, and the third path is a 7x7 convolution, with filter allocations of int(w*0.167), int(w*0.333), and int(w*0.5) respectively. Through this design, the network can capture feature information of different scales at the same time, enhancing the understanding of image details.

[0104] Edge enhanced block:

[0105] When facing the fine-scale semantic segmentation task of low semantic level, the diversity and overlap of classes are common, and there are also complex situations of same object different spectrum and different object same spectrum. These problems significantly increase the difficulty of the segmentation task, especially when dealing with weak boundaries. In order to overcome these challenges, we adopted the edge enhanced method corresponding to the ResUNet network, which was integrated into the encoder part to improve the extraction ability of edge features.

[0106] Specifically, we introduced residual blocks and edge extraction convolution blocks corresponding to multi-scale residual blocks (MultiResBlock) in the encoder part of ResUNet to capture feature information of different scales and enhance the ability to analyze details. At the same time, by embedding a lightweight attention (Efficient Channel Attention, ECA) module after the ordinary residual block and the edge extraction convolution block, the expression ability of the feature map and the edge positioning accuracy are further improved. The input of the encoder is first extracted by the backbone network, and then passes through five residual blocks in turn, with an edge enhanced EE block set after each residual block. The obtained boundary enhanced result is combined with the feature map of the MultiResBlock.

[0107] These rich features are passed through the AttentionGate to the decoder part. The AttentionGate plays an important role in this process by dynamically adjusting the weights of the feature maps, allowing the network to pay more attention to key feature regions, thereby improving the accuracy and robustness of the segmentation results. This design not only takes advantage of the edge enhancement method, but also combines the advantages of multi-scale feature extraction and attention mechanism, ensuring that the model can maintain high segmentation accuracy and stability in complex and variable input data situations.

[0108] In this way, our model can effectively deal with weak boundary phenomena and significantly improve overall segmentation performance when dealing with fine-scale segmentation tasks on high-resolution images. By combining multi-scale feature extraction, edge enhancement, and attention mechanisms, the model performs well in complex multi-target tasks, significantly improving segmentation accuracy and robustness. Through this design, the model not only performs well in high-resolution image processing, but also maintains stable segmentation effects in complex scenarios, significantly improving overall segmentation performance and application versatility.

[0109] Attention Gate (Attention-Gate):

[0110] Deep learning models trained using the Attention Gate can implicitly improve network performance, and many studies focus on binary classification tasks such as medical images and building extraction in single-class scenarios. Although the Attention Gate can further enhance the model's performance in multi-class tasks, it has not been widely used in remote sensing image segmentation. By dynamically adjusting the weights of features through the gating unit, the gated attention mechanism not only effectively enhances attention to multiple targets, but also maintains a balance of attention between classes in multi-target scenarios, thereby significantly improving overall model performance. Through this method of dynamically adjusting feature weights, the gated attention mechanism can maintain high accuracy and performance when dealing with complex multi-target tasks. The two inputs of the Attention Gate module are skip_connection, gating_signal, W x and W g The main structure consists of a 1x1 convolution and an activation function, and finally, the attention weight map is passed through an upConv_Block and a trans_conv2D operation to ensure its size matches the skip_connection.

[0111] Segmentation Network:

[0112] Unlike typical segmentation methods that consider semantic classes as disjoint labels, in high-resolution remote sensing image deep-level semantic segmentation, the potential dependency between classes is considered and not only in the form of hierarchical tree structure. In single-level semantic segmentation, each pixel is classified into a predefined semantic class without considering the dependency between classes. For an image I, the goal is to assign a label l i to each pixel i i ∈L, L is the set of all possible labels. Therefore, single-level semantic segmentation can be represented as:

[0113]

[0114] Here S(I) represents the result of single-level semantic segmentation on image I, i.e., the set of labels assigned to each pixel in the image.

[0115] In multi-level (hierarchical) semantic segmentation, the hierarchical dependency between labels is considered. Considering the hierarchical relationship between classes, we adopt a non-tree structure, such as a cobweb structure, to represent this relationship. This structure is formalized by a matrix M cf where each element W cf of the matrix represents the association weight between a coarse class c and a fine class f. The calculation of the association weight W cf is based on pixel matching statistics:

[0116] W cf = M cf [c, f] (2)

[0117] Here, M cf [c, f] represents the number of pixel matches between class c and class f, which is accumulated by iterating through all images and their corresponding labels. Then, multi-level semantic segmentation can be represented as: This representation clearly reflects the goal of multi-level semantic segmentation, i.e., finding the most suitable label for each pixel in the image under the framework of hierarchical dependency and dynamic loss adjustment.

[0118]

[0119] H * (I) represents the optimal result of multi-level semantic segmentation on image I considering the hierarchical dependency between classes and dynamic loss adjustment. is the optimal label for each pixel i obtained by solving the optimization problem.

[0120] Multi-loss synergistic regulation cobweb loss

[0121] To incorporate the hierarchical relationship between classes into the loss function, we define a dynamically adjusted loss L cobweb , which is based on the base loss Lbase and pre-computed correlation weights W cf and a dynamic factor δ, which fully considers the hierarchical dependency between classes and the dynamic changes of loss during training. I is a given image, where the goal for each pixel i is to be assigned a label l i . Our goal is to find a label assignment L = {l1, l2,..., l N} that minimizes the total loss given the image I, the correlation weight matrix W cf between classes, and the dynamic factor δ. Where H * represents the optimal label assignment, i.e., the final semantic segmentation result.

[0122] L cobweb is a loss function that takes into account the hierarchical dependency between classes, which is specifically defined as follows: L base is the traditional loss function cross-entropy loss sparse_categorical_crossentropy used to measure the difference between the true label and the predicted label of pixel i. W cf is the correlation ratio weight matrix between classes, which reflects the hierarchical dependency between classes by calculating the correlation between different classes. This structure is formalized by matrix M cf , where each element W cf of the matrix represents the correlation strength of the correlation weight between the coarse class c and the fine class f, so that when assigning labels, not only depends on the predicted value of the pixel itself, but also refers to the correlation between classes. δ is a factor dynamically calculated according to the average loss of the last few training periods. Its role is to adjust the loss function in real time during training to balance the weights of different classes in the segmentation process, preventing the model performance from declining due to class imbalance or fluctuations during training.

[0123] In the image semantic segmentation task, the optimal segmentation result H*(I) is represented as the optimal label set for each pixel i in the image I where is the optimal label assignment obtained by optimizing the loss function L cobweb . The optimization process aims to minimize the loss of each pixel, i.e., by minimizing L cobweb to determine the optimal label assignment. Specifically, the optimal label is determined by minimizing the loss of each pixel . The optimization process can be represented as formula (4) The core idea of this formula is that for each pixel i, we find the label in the possible label set L that minimizes the loss function In contrast to the general case of formula (3) Here, the optimization process explicitly selects the optimal label for each pixel by minimizing the loss function, rather than simply assigning a label. Therefore, compared to the traditional label assignment method, our optimization process not only focuses on the prediction error of a single pixel, but also adjusts the global strategy of label assignment through W cf and δ, so that the final segmentation result can better capture the hierarchical dependency between classes and improve the overall performance of the model

[0124]

[0125] We calculate the relationship ratio R of class c to class f to quantify the degree of dependence between classes:

[0126] L cobweb = L base + δ × (1 - R) (5)

[0127] When we predict from fine-grained (16 classes, denoted by f) to coarse-grained (6 classes, denoted by c), the relationship ratio R of class c to class f cf can be calculated as follows:

[0128] W cf = M cf [c, f] (6)

[0129]

[0130] Here M c，f is an element in the mapping matrix from 16 classes to 6 classes, representing the mapping strength from class f to class c. The denominator is the total number of F for a given C (coarse classification class), which ensures that the relationship ratio is normalized. At this time, the loss function is

[0131]

[0132]

[0133] N is the total number of pixels in the image. is the probability that the model predicts the nth pixel belongs to its true class y n , where y n is the true class index of the pixel, belongs to the set of fine-grained classes, ∈ is a very small constant to ensure numerical stability. is the adjusted fine-grained class loss function, is the basic loss function.

[0134] When we predict from coarse (6 classes, denoted by c) to fine (16 classes, denoted by f), the class f to class c relation ratio R fc can be calculated as follows:

[0135]

[0136] The denominator is for a given fine class f, where C represents the total number of coarse classes. The loss function at this time is:

[0137]

[0138] N is the total number of pixels in the image. is the probability that the model predicts the nth pixel belongs to its true class y n , where y n is the true class index of the pixel, belongs to the set of coarse classification categories, ∈ is a very small constant to ensure numerical stability.

[0139] Finally, taking The GID Datasets as the basis for training and evaluation, the semantic layering effect of the deep hierarchical multi-semantic data processing method in the embodiment of the present application is explained:

[0140] Dataset:

[0141] As the basis for training and evaluation, a well-annotated benchmark dataset is crucial for developing remote sensing semantic segmentation algorithms. Remote sensing semantic segmentation datasets are difficult to balance the number of data samples and the sufficiency of land cover categories. Therefore, in the growing demand of real-world applications, such as simultaneously identifying industrial land and urban residential areas, it is difficult to meet.

[0142] GID Datasets are selected as the experimental dataset, GID Datasets: including GID-5 and GID-15. GID-5 contains 150 images covering dozens of cities in China, with 5 land cover categories; GID-15 subdivides the categories of GID-5, and determines 15 related categories according to the Chinese land use classification standard.

[0143] GID-5 contains a rich data sample covering dozens of cities in China, 150 images of 6800x7200 pixels, ensuring the diversity of data for deep model training, while only 5 involving land cover categories (i.e. built-up area, farmland, forest, meadow and water) are insufficient for practical applications. The WHU GID-5 dataset is a large-scale high-resolution remote sensing image land cover dataset based on China's Gaofen-2 satellite images. The dataset consists of 150 pixel-level annotated images, including five land cover categories: buildings, farmland, forest, grassland, and water. Red for buildings, green for farmland, blue for forest, yellow for meadow, and blue for water. The image size is 6800x7200, and each image includes four bands (i.e. blue, green, red and near-infrared bands).

[0144] To propose a better benchmark that meets the actual needs, the land cover categories in GID-5 are subdivided to form a new GID-15. Specifically, to further distinguish the subcategories belonging to the same land cover general category, reference is made to the Chinese Land Use Classification Standard (GB / T21010-2017), and finally 15 related categories are determined through hierarchical classification relationships. Specifically, to study the detailed distribution of urban area, the built-up area is subdivided into industrial land, urban residential land, rural residential land and transportation land. Then, the forest is subdivided into garden land, arbor forest and shrub land for vegetation coverage study, and the meadow is subdivided into natural meadow and artificial meadow. Finally, the farmland is subdivided into paddy field, irrigated land and dry land to study the distribution of agricultural land; the water area is subdivided into river, lake and pond to study water resources.

[0145] The "Huawei Ascending Cup" remote sensing image dataset is the dataset provided in one of the tracks of the Huawei Ascending Cup competition on August 24, 2020. This dataset focuses on optical remote sensing images with typical land use classification. Based on existing remote sensing classification needs and actual needs, and referring to the geographical conditions monitoring and "three surveys" land classification standards, a feature classification system is designed that follows the "what you see is what you get" principle. This dataset contains 8 main categories and 17 secondary subcategories. It should be noted that the main categories and secondary categories are not directly corresponding to the remote sensing images. In this study, the main categories used are generated from the secondary subcategory labels defined in the official classification table.

[0146] Data source: This dataset is collected, annotated and constructed by Pengcheng Laboratory and its partners. The data sources include visible light and multispectral images from Gaofen-1, Gaofen-2, Gaofen-6, SuperView-2, Beijing-2 and some aerial images, with resolutions ranging from 0.1 meters to 4 meters.

[0147] Dataset size: This dataset includes more than 1 million semantic segmentation samples of remote sensing images.

[0148] Raw images: Images are in TIFF format, containing red (R), green (G), and blue (B) bands. The training set image size is 256x256 pixels, and the preliminary test set image size is also 256x256 pixels. The subsequent round test set image size can be variable.

[0149] Label data: Labels are stored in single-channel PNG files, with each pixel value represented by three digits. The storage format is 'uint16'. The hundreds digit represents the primary class, while the tens and units digits together represent the secondary class. A total of 100,000 primary class images and 100,000 secondary subclass images are provided. This study uses 16,000 images as the training set and 7,576 images as the validation set.

[0150] Data samples are selected from GID-5, GID-15 datasets, Huawei Ascend Cup 2020-17, and Huawei Ascend Cup 2020-8.

[0151] Evaluation indicators:

[0152] Overall accuracy (OA), precision, recall, F1 score, and mean intersection over union (MIOU) are used to evaluate model performance. These indicators are widely used in previous studies and are compared with recognized semantic segmentation evaluation standards, providing a comprehensive evaluation of the proposed model's performance.

[0153] Specifically, overall accuracy (OA) measures the proportion of correct predictions in all samples. Precision reflects the proportion of actual positives among all predicted positives. Recall represents the proportion of correctly predicted positives among all actual positives. F1 score is the harmonic mean of precision and recall, used to measure the comprehensive performance of the model in classification tasks. Mean intersection over union (MIOU) is used to evaluate the segmentation performance of the model on multiple classes. These evaluation indicators are calculated as follows: P represents the number of positive samples; N represents the number of negative samples; TP is the number of correctly predicted positive samples (True Positives); FP is the number of incorrectly predicted positive samples (False Positives); TN is the number of correctly predicted negative samples (True Negatives); and FN is the number of incorrectly predicted negative samples (False Negatives). Through comprehensive evaluation of these indicators, the performance of the proposed model in different aspects can be fully understood, allowing for more accurate judgment of its strengths and weaknesses:

[0154]

[0155] Parameter setting:

[0156] An Intel i9-7940X CPU with 8 cores and 16 threads, NVIDIA RTX3090 GPU x 2, and 64 GB of memory were used. The software environment included a 64-bit Microsoft Windows 10 operating system, Anaconda 5.2.0, CUDA 11.2, Python 3.8, and TensorFlow 2.5.0.

[0157] The transpose layer was initialized using the Xavier initialization method, and the Adam optimizer was used with a learning rate of 4.5 x 10 -4 , and the training was performed for 70 cycles with a batch size of 8.

[0158] Results and discussion:

[0159] In this part, we show the performance of the proposed method using the GID and ISPRS Vaihingen datasets and analyze its effectiveness.

[0160] Performance of HAGNet network:

[0161] In the L cobweb loss function we proposed, the class transition matrix was used to effectively capture the specific relationship between classes and dynamically designed into the loss function, forming a strong constraint between multiple labels. Compared with traditional methods, the L cobweb showed significant advantages in multiple models and indicators. Table 1 shows the classification ability of HAGNet in coarse class features. From the data, the L cobweb loss function improved the performance of the Deeplabv3+ model compared to the L base loss function. For example, the IoU of the meadow (grass) class improved to 77.05%, which was significantly better than the 72.87% of the L base loss function. At the same time, the mIoU and FWIoU indicators also improved, reaching 88.85% and 87.83%, respectively. These data reflect that the L cobweb performed more stably on complex classes, and its dynamic adjustment mechanism can effectively handle subtle differences in multi-class segmentation, avoiding the limitations of traditional single constraint criteria. In the MAE-BG model, the L cobweb also significantly improved the performance of multiple classes, especially in the water class, with an IoU of 93.42%, which was significantly higher than the 86.38% of the L base loss function, demonstrating the advantages of the L cobweb in handling difficult segmentation classes. Overall, the L cobwebmIoU and FWIoU are improved to 90.41% and 91.38% respectively, showing its robustness among different classes. In our proposed HAG model, the IoU of farmland and water classes reaches 97.26% and 96.06% respectively, compared with L base , which has improved in both, further verifying the effectiveness of its dynamic loss adjustment mechanism. Overall, L cobweb By dynamically designing the reasoning relationship and class transition matrix, the hierarchical structure is encoded into the pixel embedding space, ensuring high accuracy of the segmentation task in complex scenes.

[0162] In Table 2, we can see that L cobweb loss function has shown significant advantages in each model. Compared with L base , L cobweb uses a class transition matrix to dynamically adjust the loss function by capturing the hierarchical relationship between classes, thereby achieving more fine-grained constraints on multi-class segmentation tasks. This method not only improves the adaptability of the model to various scenes, but also significantly improves the segmentation accuracy of different classes.

[0163] In the Deeplabv3+ model, L cobweb reduces classification errors in complex scenes, making the overall accuracy improve to 79.93%, with IoU in Garden and River classes reaching 90.23% and 80.19% respectively, significantly better than L base method. This shows that L cobweb can better capture the subtle differences between classes and enhance the model's segmentation ability in complex scenes. For the MAE-BG model, L cobweb significantly improves the IoU in multiple classes, such as Urban land reaching 80.53% and Dry crop reaching 85.95%. Compared with L base method, L cobweb more effectively handles the complex relationships between class hierarchies, making the overall accuracy reach 83.39%, showing strong generalization ability, especially when dealing with multi-class segmentation tasks, the performance is more stable. In our proposed HAG model, the introduction of L cobweb loss function makes the model achieve significant improvement in multiple classes, especially in the IoU of Pond class from 18.46% to 24.02%, which is particularly outstanding. This further shows that through the dynamic adjustment of the class transition matrix, the model can better capture the complex dependency relationships between classes, effectively improving the model's performance in various complex scenes.

[0164] In summary, L cobwebThe loss function significantly enhances the precision and robustness of the model in processing the multi-class segmentation task by dynamically adjusting the hierarchical relationship between classes. Compared with the simple superposition of L base , L cobweb provides a more flexible and effective segmentation strategy, especially for high-resolution remote sensing image segmentation tasks that need to handle complex class associations.

[0165] Table 1 shows that HAG exhibits superior performance in high semantic classes, especially in the built-up, farmland, and meadow classes, showing superior performance and excellent segmentation ability. In the built-up class, HAG achieved an accuracy of 99.07 and a recall rate of 98.27, which is 1.51 higher in accuracy and 1.08 higher in recall rate than the worst segmentation network CMLF. In the farmland class, HAG's accuracy is 96.87, and the recall rate is 97.64, which is 2.65 higher in accuracy and 4.25 higher in recall rate than the worst SUNet. For the meadow class, HAG performs best with an accuracy of 99.08 and a recall rate of 96.31, which is 1.91 higher in accuracy and 1.25 higher in recall rate than CGCL. According to Table 2, HAG achieves an F1 score of 98.66 and an IoU of 97.26 in the built-up class, surpassing the worst-performing SUNet by 0.76 in F1 score and 1.37 in IoU. In the farmland class, HAG's F1 score is 96.57, and the IoU is 91.73, which is 2.77 higher in F1 score and 15.84 higher in IoU than SUNet. In the meadow class, HAG achieves an F1 score of 98.17 and an IoU of 96.06, which is 2.40 higher in F1 score and 4.17 higher in IoU than SUNet. Compared to the optimal SFFNet, HAG is only 0.77 lower in built-up class IoU but 0.88 higher in farmland class F1 score. This accuracy not only surpasses the worst-performing network SUNet by 0.76 in F1 score and 1.37 in IoU but also outperforms the best-performing network SFFNet by 0.88 in farmland class F1 score. According to the experimental results in Tables 3 and 4, our method (HAG) exhibits superior performance in semantic segmentation on the GID dataset, especially showing significant advantages in some high semantic classes. In the Urb.land class, HAG achieves an accuracy of 88.21 and an IoU of 77.9, which is 10.92 higher in accuracy and 11.19 higher in IoU than the worst-performing network SUNet. In the Traff.land class, HAG achieves an accuracy of 78.5 and an IoU of 66.14, which is 55.42 higher in accuracy and 48.07 higher in IoU than SUNet, demonstrating its strong ability to capture complex traffic features. In the Irr.land class, HAG achieves an accuracy of 91.21 and an IoU of 87.53, which is significantly higher than the relatively poor Deeplabv3+, with an accuracy of 12.4 and an IoU of 15.88. Furthermore, in the Nat.mead class, HAG achieves an accuracy of 95.04 and an IoU of 68.03, which is 31.76 higher in accuracy and 17.01 higher in IoU than SUNet.Compared with the best-performing SFFNet, HAG still demonstrates competitiveness in some categories such as urban land and transportation land, and its performance is slightly lower than SFFNet in irrigation land and natural grassland categories, but it still maintains strong performance in other categories. Overall, HAG demonstrates excellent stability and accuracy in multi-category scenarios, especially in complex and high-semantic categories, verifying its advantages as an efficient and reliable semantic segmentation method on the GID dataset. Figure 3 As shown in the segmentation results of the GID test set in the above network, Deeplabv3+ relies on the Atrous Spatial Pyramid Pooling (ASPP) module to better capture multi-scale context information, but the segmentation results are slightly rough in areas with complex details. The CGGL network guides global context modeling through class information, and performs stably, but has slightly insufficient detail capture in some edge regions. The MAE-BG network effectively enhances the detail segmentation capability through edge detection and boundary guidance, and exhibits excellent edge continuity. Although SUNet combines Swin Transformer and UNet architecture, it has good context modeling capability, but its segmentation accuracy in complex regions is not as ideal as other networks. CMLF exhibits strong feature extraction capability and detail retention, suitable for large-scale region segmentation, but there is still room for improvement in edge processing. SFFNet makes full use of spatial information and performs well in complex regions and edge processing, maintaining high segmentation accuracy. Most notably, HAG Net performs the most outstandingly among all the comparison networks, especially in complex regions and high-semantic categories (such as roads and buildings), exhibiting excellent detail capture and boundary processing capability. The hierarchical semantic segmentation results of HAG maintain consistency and high accuracy in multi-category scenarios, verifying its advantages as an efficient and stable semantic segmentation method.

[0166] According to the experimental results and network structure analysis of Tables 5 and 6, the HAG network shows outstanding advantages in the Water, Traffic, and Farmland categories. The HAG achieved an F1 score of 74.60 and an IoU of 59.49 in the Water category, which is 31.26 higher in F1 score and 31.83 higher in IoU than the worst-performing CMLF network. In the Traffic category, the HAG's F1 score is 81.77, and the IoU is 69.16, which is 38.43 higher in F1 score and 41.50 higher in IoU than the worst-performing CMLF. In the Farmland category, the HAG's F1 score is 81.94, and the IoU is 69.41, which is 59.73 higher in F1 score and 56.92 higher in IoU than the worst-performing CGCL. Compared with the best-performing network such as SFFNet, the HAG is 21.17 higher in IoU in the Water category but slightly lower by 0.40 in F1 score in the Traffic category. In addition, the HAG's performance in F1 score in the Farmland category is close to the best-performing network, and the IoU is slightly improved.

[0167] According to the results in the table and the performance of each network, the HAG network shows outstanding segmentation ability in multiple categories, especially in the Photovoltaic, Road, and Water categories. In the Photovoltaic category, the HAG achieved an F1 score of 84.00 and an IoU of 73.19, which is 15.17 higher in F1 score and 5.91 higher in IoU than the worst-performing CMLF network. In the Road category, the HAG's F1 score is 79.60, and the IoU is 61.83, which is 18.90 higher in F1 score and 15.90 higher in IoU than the worst-performing CGGL network. In the Water category, the HAG's F1 score is 77.88, and the IoU is 66.64, which is 56.86 higher in F1 score and 45.62 higher in IoU than the CMLF network. Compared with the best-performing network such as SFFNet, the HAG is slightly higher by 0.16 in IoU in the Photovoltaic category but lower by 6.64 in F1 score in the Water category. In the Road category, the HAG's performance is close to SFFNet, with an F1 score that is 0.61 higher. The outstanding performance of the HAG in the Photovoltaic, Road, and Water categories demonstrates its excellent ability to handle details and capture contextual information in complex scenes. From Figure 4It can be seen that different networks show obvious differences in the semantic segmentation task of complex scenes. Deeplabv3+ performs well in homogeneous regions, but has more misclassification and discontinuity in complex boundaries and detail regions. CGGL performs poorly in handling fine boundaries in shengtengcups, and the figure shows that it is slightly weak in capturing fine structures. The MAE-BG network is more prominent in edge detection and detail preservation, and the segmentation result is more complete when dealing with complex edge regions, but its effect still has room for improvement for very small features. SUNet uses the characteristics of Swin Transformer to perform stable context modeling, but it is prone to segmentation discontinuity and classification errors in detail-rich regions. CMLF performs well in large-area segmentation, but it is slightly insufficient in handling complex boundaries, and the figure shows that its performance in detail is slightly weak. SFFNet performs well in utilizing spatial information and handling complex regions, and the edge processing is relatively clear, but the segmentation accuracy is not as good as HAGNet in extremely complex scenes. HAG Net exhibits excellent segmentation performance, especially in handling high-complexity regions, with clear segmentation edges, complete detail preservation, and effective reduction of classification errors. It is superior to other networks in multi-class distinction and detail capture, verifying its stability and accuracy as an efficient semantic segmentation method in complex scenes.

[0168] In the ablation experiment section, we will show two aspects in detail: the ablation experiment of the cobweb loss function and the ablation experiment analysis of each module in the HAG network in the GID dataset. First, we explore the impact of the cobweb loss (L_cobweb) on the overall model performance. L_cobweb is designed to enhance the ability of inter-class collaboration and interaction modeling, so as to more accurately capture the complex structure and semantic relationship of ground objects. By removing this loss term and conducting experiments for comparison, we can intuitively observe the contribution of L_cobweb to the segmentation result. This study selects the classic Deeplabv3+ network and the MAE-BG network based on edge algorithm, and respectively evaluates the performance after adding L_cobweb, as shown in Figure 5

[0169] ​In the GID-5 class test set, different networks all show significant improvement after using L_cobweb. After adding L_cobweb, the overall accuracy (OA) of Deeplabv3+ increases by 1.85%, reaching 94.04%; the mIoU increases by 3.08%, reaching 90.30%; and the FWIoU increases by 3.08%, reaching 88.86%. In the class analysis, the highest improvement is in the "water" class, with an OA increase of 4.87%, and mIoU and FWIoU increases of 6.77%. In the MAE-BG network, L_cobweb increases the OA from 93.70% to 95.48%, an increase of 1.78%; the mIoU increases by 0.65%, reaching 90.41%; and the FWIoU increases by 3.02%, reaching 91.38%. Among them, "water" has the highest improvement in OA, mIoU, and FWIoU, with increases of 6.92% and 7.82%, respectively. In the HAG network, after using L_cobweb, the OA increases by 1.26%, reaching 96.74%; the mIoU increases by 2.87%, reaching 93.86%; and the FWIoU increases by 2.39%, reaching 93.73%. In the class analysis, the highest OA improvement is in the "built-up" class, with an increase of 2.16%; the highest mIoU and FWIoU improvements are in the "meadow" class, with increases of 5.73%. In the GID-15 class test set, L_cobweb also significantly improves the performance of each network. In Deeplabv3+, the OA increases by 4.12%, reaching 82.24%; the mIoU increases by 0.39%, reaching 48.51%, although the increase is small; and the FWIoU increases by 2.31%, reaching 70.13%. Among them, the highest improvement is in the "Urb.land" class, with an OA increase of 5.07%; and the "Pond" class has the highest mIoU and FWIoU improvements, with increases of 1.91%. In the MAE-BG network, after using L_cobweb, the OA increases by 3.75%, reaching 83.39%; the mIoU increases by 2.18%, reaching 54.34%; and the FWIoU increases by 4.68%, reaching 75.17%. The highest improvement is in the "P.field" class, with an OA increase of 9.27%; and the "Garden" class has the highest mIoU and FWIoU improvements, with increases of 3.75%. For the HAG network, after using L_cobweb, the OA increases by 3.43%, reaching 86.94%; the mIoU increases by 4.20%, reaching 58.92%; and the FWIoU increases by 4.47%, reaching 79.31%. The class analysis shows that "Urb.land" has the highest improvement in OA, mIoU, and FWIoU, with increases of 11.48% and 9.97%, respectively.These results verify the effectiveness of the L cobweb loss function in different networks, not only significantly improving the classification performance, but also enhancing the segmentation ability and overall performance of the model in complex classes and large-scale scenarios.

[0170] In Shengteng Cup-8 test set, L_"cobweb" loss function brings significant improvement in different networks. For Deeplabv3+ network, after using L_"cobweb", the overall accuracy (OA) increases by 3.40%, reaching 74.52%; mIoU rises by 3.16%, reaching 56.89%; and FWIoU improves by 4.17%, reaching 59.69%. In class analysis, the highest improvement class is "Water", with OA, mIoU and FWIoU increasing by 3.98% and 6.06% respectively. In MAE-BG network, after introducing L_"cobweb", OA increases by 1.47%, reaching 75.42%; mIoU increases by 4.18%, reaching 56.65%; and FWIoU improves by 1.73%, reaching 60.85%. The highest improvement class is "Traffic", with OA increasing by 3.43%, while mIoU and FWIoU increase by 1.64%. For HAG network, after using L_"cobweb", OA increases by 0.84%, reaching 75.42%; mIoU increases by 1.00%, reaching 56.65%; and FWIoU improves by 1.22%, reaching 60.85%. Among them, the class with the highest improvement is "Bare Soil", with OA increasing by 4.75%, while the class with the highest mIoU improvement is "Traffic", increasing by 4.67%. In Shengteng Cup-17 test set, the application of L_"cobweb" loss function in different networks also brings obvious improvement. For Deeplabv3+, OA increases by 3.89%, reaching 70.88%; mIoU increases by 3.83%, reaching 48.08%; and FWIoU improves by 4.67%, reaching 55.14%. The class with the highest improvement is "Photovoltaic", with OA and mIoU increasing by 6.85% and 9.44% respectively. In MAE-BG network, after using L_"cobweb", OA increases by 1.64%, reaching 71.73%; mIoU increases by 2.96%, reaching 51.85%; and FWIoU increases by 1.75%, reaching 56.14%. In this network, the class with the highest improvement is "Parking lot", with OA increasing by 7.01% and mIoU increasing by 3.21%. For HAG network, L_"cobweb" loss function makes OA increase by 3.12%, reaching 73.67%; mIoU increases by 3.47%, reaching 54.58%; and FWIoU improves by 3.87%, reaching 58.60%. Among them, the class with the highest improvement is "Photovoltaic", with OA increasing by 3.30% and mIoU increasing by 1.47%.These results validate the effectiveness of the L_"cobweb" loss function in improving the classification ability of different networks and the overall segmentation performance.

[0171] In the experiments on the GID dataset, we conducted ablation experiments on the key modules introduced by the HAG network to analyze their impact on the overall network performance. Specifically, this part of the experiment includes removing or replacing the multi-scale feature extraction module, attention mechanism module, and class relationship modeling module in the HAG network, and observing their impact on the accuracy, IoU, and F1 score of the ground object segmentation. By gradually removing these modules, we can determine the role of each module in capturing contextual information, enhancing semantic consistency, and improving the segmentation performance of complex regions. The baseline model is ResUNet, the ED module combines Residual block and ECA module, MRB is multi-scale Residual block, GA-Block is a gated attention mechanism module, and Lcobweb is a dynamic cobweb loss module. In the GID-5 class test set, the performance of the HAG network gradually improves with the gradual introduction of modules. After adding the ED module, the overall accuracy (OA) increases to 95.48%, mIoU and FWIoU also increase to 90.99% and 91.34%, respectively. After introducing the MRB module, OA further increases to 96.14%, mIoU and FWIoU increase to 92.85% and 92.59%, respectively. When the GA-Block is added, although OA decreases slightly to 96.01%, the overall performance remains stable, and it is proven that the GA-Block is more suitable for multi-class classification. Finally, when the Cobweb module is introduced into the complete configuration, OA increases to 96.74%, mIoU and FWIoU reach 93.86% and 93.73%, respectively. In the GID-15 class test set, the gradual addition of different modules also brings significant improvements. After adding the ED module, OA increases to 83.52%, mIoU and FWIoU increase to 54.72% and 74.84%, respectively. After introducing the MRB module, OA increases to 84.24%, mIoU and FWIoU reach 55.53% and 76.34%, respectively. The addition of GA-Block further increases OA to 85.07%, and FWIoU increases slightly to 76.69%. After adding the Cobweb module in the complete configuration, OA increases significantly to 86.94%, mIoU reaches 58.92%, and FWIoU increases to 79.31%. Overall, in the GID-5 class test set, the complete configuration of the HAG network improves OA by 1.49%, mIoU by 2.89%, and FWIoU by 2.68% compared to the baseline. In the GID-15 class test set, the complete configuration of the HAG network improves OA by 4.47%, mIoU by 4.91%, and FWIoU by 5.64%.These results show that the step-by-step introduction of ED, MRB, GA-Block, and Cobweb modules can significantly improve the classification performance of the HAG network on the GID dataset, especially in the full configuration, where the network exhibits the highest precision and consistency.

[0172] Conclusion: By introducing hierarchical dependencies between classes and a dynamic loss adjustment mechanism, the deep semantic segmentation method proposed in this paper shows significant advantages in high-resolution remote sensing image segmentation tasks. This method uses a special class transition matrix to capture specific relationships between classes, dynamically integrating them into the loss function to achieve forced constraints and collaborative optimization between multiple classes. This makes the segmentation process not only efficient in capturing contextual information, but also highly precise and stable in complex background and multi-level class relationship scenarios.

[0173] Experimental results show that our method performs outstandingly in both coarse-grained and fine-grained class segmentation tasks, demonstrating its exceptional ability in fine classification and complex scene processing. Especially by introducing a dynamic loss adjustment mechanism, our method has reached a new level in class relationship modeling and multi-scale feature fusion, effectively reducing segmentation errors and enhancing edge processing clarity. This innovative interactive optimization process allows the relationships between ancestors and child classes to influence each other during the segmentation process, thereby improving overall segmentation performance.

[0174] In addition, experimental results further verify the key role of each module in improving segmentation performance, demonstrating the unique contribution of the Cobweb loss in capturing the structure of ground objects and class interaction relationships. Future work will focus on exploring more complex class hierarchies and optimizing dynamic loss adjustment strategies to further improve segmentation accuracy and model generalization. We also plan to extend the method to more types of remote sensing image data to verify its applicability in different environments and application scenarios. Through continuous improvement, we expect to continue to achieve higher classification accuracy and model stability in handling more complex and diverse remote sensing tasks.

[0175] Comparison with classic models:

[0176] The visualizations of our network in remote sensing image segmentation, especially under the influence of the Cobweb loss we proposed, show several significant improvements. First, for the farmland part, traditional segmentation methods often struggle with different spectral color changes due to seasonal variations, making it difficult to accurately identify farmland areas. However, with the introduction of the Cobweb loss, our network successfully overcomes this challenge, achieving precise segmentation of farmland areas. Second, the water body area shows clearer edge positioning after processing by our network, significantly reducing misclassification problems between buildings and water bodies, grasslands, and other categories. This is because the Cobweb loss effectively understands and utilizes the hierarchical relationships between categories and boundary constraints, thereby improving the model's ability to handle edge details.

[0177] In terms of fine-class segmentation, especially for the road area, traditional annotation often labels surrounding areas as roads to simplify operations, leading to blurred boundaries during segmentation. Our network, through the interaction of the Cobweb loss and six-class information, not only accurately divides the road area but also effectively distinguishes adjacent non-road areas, significantly improving the classification accuracy of Irrigate land and Traffic land. This advantage stems from our network's ability to model class-level dependencies, better handling complex land types and providing more accurate semantic segmentation results.

[0178] The classic network deeplabv3+ has a small improvement after adding the Cobweb loss, and compared with the results of other networks after adding the Cobweb loss, it is found that the deeplabv3+ due to the basic classification confusion caused by the empty convolution in the empty spatial pyramid pooling, the constraint of the Cobweb loss cannot take good effect. Second, the law improves, the whole data set, some categories decrease, not in line with the overall law of some samples in the test set, by introducing the hierarchical relationship between classes and the dynamic loss adjustment mechanism, the deep semantic segmentation method proposed in this paper shows excellent segmentation performance on high-resolution remote sensing images, especially in handling scenes with complex backgrounds and multi-level class relationships. Future work will explore more complex class hierarchy structures and loss adjustment strategies to further improve segmentation accuracy and model generalization ability.

[0179] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the present application can be realized by a general computer device, which can be concentrated on a single computer device or distributed on a network composed of multiple computer devices, and optionally, each module or each step can be realized by program codes executable by a computer device, so that each module or each step can be stored in a storage device and executed by a computer device, and in some cases, the steps shown or described can be executed in different order, or each module or each step can be manufactured into an integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Therefore, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0180] It should be noted that the above is only the preferred embodiment of the present application, and does not limit the patent protection scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

[0181] Table 1. Evaluation of GID-5 test set on OA, Recall

[0182]

[0183] Table 2. Evaluation of GID-5 test set on F1, MIOU, FWIOU

[0184]

[0185] Table 3. Evaluation of GID-15 test set on precision

[0186]

[0187]

[0188] Table 4. Evaluation of GID-15 dataset on IOU

[0189]

[0190] Table 5. Evaluation of Sheng teng Cup-8 test set on precision and recall

[0191]

[0192]

[0193] Table 6. Evaluation of Sheng teng Cup-8 test set on F1, IOU and FWIOU

[0194]

[0195] Table 7. Evaluation results of Sheng teng Cup-17 test set in precision and F1 value

[0196]

[0197]

[0198] Table 8. Evaluation results of Sheng teng Cup-17 test set in precision and IOU and FWIOU value

[0199]

[0200] Table 9. Evaluation of HAG network on GID test set (coarse categories)

[0201]

[0202] Table 10. Evaluation of HAG network on GID test set (fine categories)

[0203]

[0204] In the above Tables 1-10, two data sets are included, which are: GID test set and Sheng teng Cup test set.

[0205] In which the 6 Chinese labels in GID-5 test set are: Building (built-up): building, Farmland: farmland, Forest (forest), Meadow (meadow), Water (water area). The 15 Chinese labels in GID-15 test set are: Industrial land (industrial land), Urban residential (urban residential area), Rural residential (rural residential area), Traffic land (traffic land), Paddy field (paddy field), Irrigated land (irrigated land), Dry cropland (dry cropland), Garden plot (garden plot), Arbor woodland (arbor woodland), Shrub land (shrub land), Natural grassland (natural grassland), Artificial grassland (artificial grassland), River (river), Lake (lake), Pond (pond).

[0206] Similarly, the 7 Chinese labels in the Sheng teng Cup-8 test set are: Water, Traffic, Farmland, Grassland, Woodland, Bare Soil, and Others. The 17 Chinese labels in the Sheng teng Cup-17 test set are: Water, Road, Airport, Train Station, Building, SolarPanel, Parking Lot, Playground, Farmland, Greenhouse, Natural Grassland, Lawn, Natural Forest, ArtificialForest, Natural Bare Soil, Artificial Bare Soil, and Others.

Claims

1. A deep hierarchical multi-semantic data processing method based on the collaboration of dynamic spider webs and loss functions, characterized in that, This deep hierarchical multi-semantic data processing method is applicable to the hierarchical semantic segmentation network HAG Net model built on the ResUNet network with an encoder-decoder structure. Its encoder part includes multi-scale residual blocks and edge enhancement modules, its bridge part adopts ordinary convolutional network and hollow spatial pyramid pooling ASPP module, and its decoder part consists of multi-scale residual blocks and attention gates. This deep hierarchical multi-semantic data processing method includes: In the encoder part, feature information at different scales is captured by multi-scale residual blocks, and the potential hierarchical dependency between semantic categories is represented by a spider web structure. The weights are associated in a matrix formalization. The edge enhancement module is used to lift and enhance the edge information of the details, thereby improving the ability to recognize the target boundary. In the bridge section, different sizes of dilated convolutional kernels in the ASPP module are used to capture global contextual information at different scales, thereby improving the model's global feature capabilities. In the decoder section, multi-scale residual blocks are used to recover the spatial resolution of the feature map layer by layer to extract feature information, and then the feature information is concatenated with the edge features from the encoder through an attention gate. The spider web loss is adjusted collaboratively using a dynamically adjusted optimized loss function. The optimized loss function is constructed based on the base loss function and pre-calculated association weights. The association weight matrix reflects the hierarchical dependency between semantic categories and is used to balance the weights of different semantic categories in the segmentation process.

2. The deep hierarchical multi-semantic data processing method according to claim 1, characterized in that, The matrix is ​​denoted as The association weight is denoted as Through matrix To formalize association weights ; Each element of the matrix Indicates coarse classification with subcategories The correlation weight between them; Association weight The calculation is based on pixel matching statistics: ,in, Indicates category With category The number of pixel matches between images is obtained by traversing all images and their corresponding labels and accumulating the results.

3. The deep hierarchical multi-semantic data processing method according to claim 2, characterized in that, The optimized loss function is denoted as The basic loss function is denoted as The dynamic factor is denoted as The optimized loss function is expressed as: Among them, the ratio of relationships This is a quantified value that measures the degree of dependency between semantic categories. In image semantic segmentation tasks, the optimal segmentation result H∗(I) is represented as the image... Each pixel The optimal set of tags ,in By optimizing the loss function The optimal label that minimizes the loss function is obtained; The optimization process for image semantic segmentation tasks can be represented as follows: ； In the formula, Represents pixels The true label; Represents pixels Predicted labels.

4. The deep hierarchical multi-semantic data processing method according to claim 3, characterized in that, When from subcategories To coarse classification When making predictions, the category For category Relationship ratio The calculation method is as follows: , In the formula, From subcategories To coarse classification The elements in the mapping matrix represent categories. To Category The mapping strength, where the denominator is a coarse classification given the given classification. , in This represents the total number of subcategories, ensuring that the relationship ratio is normalized; At this point, the base loss function and the adjusted sub-category loss function are respectively: ,and In the formula, It is the total number of pixels in the image. The model predicts the first Each pixel belongs to its true category. The probability, This is the true category index of the pixel, belonging to the subdivision category set. It is a very small constant used to ensure numerical stability.

5. The deep hierarchical multi-semantic data processing method according to claim 3, characterized in that, When classifying by coarseness To subcategories When making predictions, the category For category Relationship ratio The calculation method is as follows: , In the formula, From subcategories To coarse classification The elements in the mapping matrix represent the subcategories. To coarse classification The mapping strength, where the denominator is the value for a given subcategories. ,in Represents the total number of coarse categories; At this point, the base loss function and the adjusted sub-category loss function are respectively: and, In the formula, It is the total number of pixels in the image. The model predicts the first Each pixel belongs to its true category. The probability, This is the true class index of the pixel, belonging to the coarse class set. It is a very small constant used to ensure numerical stability.

6. The deep hierarchical multi-semantic data processing method according to any one of claims 2, 4, and 5, characterized in that, There are 6 broad categories and 16 detailed categories.

7. The deep hierarchical multi-semantic data processing method according to claim 1, characterized in that, In the encoder section, the multi-scale residual block includes three parallel convolutional paths, 3x3, 5x5 and 7x7 convolutions, which capture feature information at different scales through multi-resolution paths and enhance overall performance.

8. The deep hierarchical multi-semantic data processing method according to claim 1, characterized in that, There are four edge enhancement modules; the edge enhancement modules are obtained by embedding lightweight attention ECA modules after the ordinary residual block and the edge extraction convolution block. In the decoding part, multi-scale residual blocks are used to restore the spatial resolution of the feature map layer by layer, which involves a total of five decoding steps. During each layer of decoding, the current feature map is concatenated with the corresponding skip connection feature map and enhanced by an attention gate to highlight important feature regions. Each layer of the decoder also concatenates the feature map of the current layer with the edge features from the encoder and the feature information from the multi-scale residual block (MultiRes Block), thereby further fusing multi-scale feature information.

Images