Point annotation weakly supervised semantic segmentation method

Abstract

The application discloses a kind of point annotation weak supervision's semantic segmentation method, including steps S1, model hyperparameter setting;S2, data preparation and normalization;S3, based on the multi-scale feature coding of SGF-Encoder;S4, based on the multi-level feature fusion decoding of HSA-Decoder;S5, the setting of loss and training strategy.The application provides a kind of point annotation weak supervision's semantic segmentation method, fuses the priori of edge / bilateral filtering, morphology, wavelet in traditional image processing with deep network structure, forms the encoder (SGF-Encoder) of " spectral local-global context-filtering priori " joint feature learning, and cooperates hierarchical structure perception decoder (HSA-Decoder) to carry out multi-scale alignment and structure reconstruction, to further under the condition of only about 0.01% pixel point annotation, obtain higher precision and significantly improve explainability and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a point annotation weakly supervised semantic segmentation method. BACKGROUND

[0002] Remote sensing image semantic segmentation aims to assign semantic categories to each pixel, which is an important underlying technology in the application fields of land cover monitoring, urban fine mapping, ecological environment assessment, disaster emergency, and traffic facility feature extraction. With the diversification of acquisition means and imaging carriers, multi-source data such as high-resolution, multi-spectral, panchromatic, and SAR are constantly enriched, image resolution is rapidly improved from meter level to sub-meter and centimeter level, scene details and texture complexity are significantly increased, and traditional methods based on handcrafted features and shallow classifiers have difficulty in meeting the actual needs in terms of class separability, boundary description, and robustness. In recent years, although deep convolutional networks and Transformer structures have made significant progress under full supervision, their dependence on large-scale pixel-level accurate labeling leads to high cost, limited cross-regional generalization, and delayed updates, making it difficult to adapt to rapidly changing ground environments and engineering deployment pace.

[0003] To reduce the labeling cost, weakly supervised learning has gradually become a research hotspot. Weakly supervised label forms include image-level labels, block-level labels, coarse labels (such as polygons, line segments), a small number of key points / point annotations, etc. Among them, point annotation is concerned due to its high labeling efficiency, convenient scene migration, and ease of crowdsourcing or semi-automatic generation. However, point annotation is naturally sparse, and the supervision signal is mainly concentrated in a small number of confirmed positive samples, and non-labeled pixels lack direct constraints, which can cause the following problems in the model: (1) incomplete boundaries: it is difficult for sparse points to propagate to complete contours in complex boundary regions such as thin lines, broken lines, and holes, leading to the appearance of artifacts such as “adhesion”, “breakage”, and “edge burrs”; (2) internal unevenness: texture differences, shadows, and sensor noise within the same region cause prediction patching and hole formation; (3) cross-class confusion: spectral-texture similar classes such as roads / rooftops, impervious surfaces / bare land, and low vegetation / cultivated land are easily eroded under point-only supervision; (4) lack of explainability: the intermediate representation of end-to-end deep networks is difficult to intuitively correspond to geometric boundaries and spectral features, making engineering tuning and scene migration costly.

[0004] The common idea of the existing weak supervision method can be summarized into three categories. The first category: based on region growing or pseudo-label generation, the point annotation is expanded to dense supervision through graph-based propagation, superpixel consistency, CRF / dense conditional random field post-processing, etc.; but this kind of method is sensitive to hyperparameters, and is easy to appear error expansion in noise, occlusion and complex structure scene. The second category: emphasizes network structure and loss design, introduces attention, boundary branch, prototype contrast learning, confidence constraint, region consistency and sparse regularization in the backbone network to alleviate the drift caused by sparse supervision; however, if there is a lack of explicit boundary and frequency domain prior, the network may still fall into the local optimum of "texture equivalence". The third category: focuses on the fusion of multi-source information and prior, such as the complement of optical and SAR, the use of multi-scale morphology, the introduction of a learnable filter or frequency domain transformation, in order to enhance the structure perception and denoising ability without dense labels; but the existing scheme often simply connects or post-processes the prior and deep features, and does not realize the integrated collaborative optimization of "prior-feature-decoding" in an end-to-end framework, so that the value of the prior is difficult to fully release.

[0005] At the specific challenge level, first is the boundary preservation problem; high-resolution images contain a large number of slender objects (roads, rivers, roof edges, fences, etc.), and traditional downsampling will lose gradient and edge intensity at a shallow level; at the same time, point supervision is difficult to cover the key points of the boundary, resulting in insufficient information in the upsampling recovery stage. Second is the noise and texture interference; shadows, vegetation lines, repeated roofs, etc. introduce high-frequency artifacts, and if there is a lack of explicit modeling and selective suppression of high-frequency / low-frequency, the network is easy to overfit local texture, causing uneven prediction within the region. Third is the lack of global context; it is difficult to establish the "shape-structure" consistency across the building group and street scale by relying on local convolution alone; although windowed attention can capture long-range dependencies, under sparse supervision, the learning of attention weights is easy to be dragged by noise, and an interpretable boundary / frequency domain constraint needs to be introduced for "correction". In addition, the loss design is difficult due to the sparsity of supervision and the imbalance of positive and negative: how to reliably converge at the point and at the same time apply a mild and stable consistency / prior constraint to the unlabeled area is the key to whether the weak supervision method can be put into practice. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a point annotation weak supervision semantic segmentation method, which integrates the prior of edge / bilateral filtering, morphology and wavelet in traditional image processing with deep network structure to form a joint feature learning encoder (SGF-Encoder) of "spectral local-global context-filtering prior", and cooperates with a hierarchical structure perception decoder (HSA-Decoder) for multi-scale alignment and structure reconstruction, thereby achieving higher accuracy and significantly improving interpretability and robustness under the condition of only about 0.01% pixel point annotation.

[0007] A point annotation weakly supervised semantic segmentation method, comprising the following steps:

[0008] S1, model hyperparameter setting;

[0009] S1-1: basic channel construction and stage expansion;

[0010] S1-2: using self-attention structure;

[0011] S1-3: training and inference setting;

[0012] S2, data preparation and normalization;

[0013] S2-1: remote sensing image cropping and data division;

[0014] S2-2: normalization and alignment;

[0015] S3, multi-scale feature coding based on SGF-Encoder;

[0016] S3-1: shallow embedding;

[0017] S3-2: three-branch joint feature learning through SGF-FL module;

[0018] S3-3: wavelet preserving down-sampling through WPD module;

[0019] S3-4: multi-stage progressive coding;

[0020] S3-5: deep semantic refinement;

[0021] S4, multi-level feature fusion decoding based on HSA-Decoder;

[0022] S4-1: shallow edge enhancement and denoising;

[0023] S4-2: overlapping patch embedding is adopted for middle-deep layer features to realize cross-stage semantic alignment and context modeling;

[0024] S4-3, multi-scale splicing and structure perception fusion;

[0025] S4-4: up-sampling reconstruction and classification output;

[0026] S5, loss and training strategy setting.

[0027] In step S1-1, the number of shallowest layer feature channels C=32 is taken as the starting point to construct a multi-stage encoder, and the spatial resolution of the feature map is reduced by 1 / 2 and the channel number is expanded by 2 times for each next stage, thereby forming a hierarchical pyramid of "resolution reduction and channel promotion";

[0028] In step S1-2, in order to establish a long-distance dependency relationship at a lower complexity, a windowed multi-head self-attention structure is adopted in the Transformer feature extraction layer of the multi-stage encoder; wherein 8 attention heads are fixedly arranged in each of the Transformer feature extraction layers, and the window side length is 8;

[0029] In step S1-3, in the training stage, an Adam optimizer is adopted, the initial learning rate is 0.001, the weight decay is 1e-4, and the learning rate is gradually reduced by combining a cosine annealing strategy to obtain smooth convergence, and a cross-entropy loss is used as a model optimization loss calculation method.

[0030] In step S2-1, in order to balance the memory occupation and scene integrity, the original remote sensing image is cut into a 256x256 patch, or the original remote sensing image is cut into a 512x512 patch when resources permit, the patch is divided into a training set, a validation set and a test set according to a 6:2:2 ratio, and the patches in each data set are labeled by using a point annotation method;

[0031] The point annotation is randomly sampled and labeled in the effective area corresponding to different categories, and the total annotation pixel ratio is 0.01%, and the number of categories numclass is set for each data set; in order to reduce the training bias, it is necessary to pay attention to maintaining the balance between different categories during the sampling process;

[0032] In step S2-2, the input patch is normalized in the channel dimension to eliminate the overall brightness and contrast shift caused by the imaging condition difference; if necessary, global brightness / contrast normalization or histogram matching can be further adopted to alleviate the domain difference of cross-scene and cross-time data; finally, the training sample image X0∈R H×W×3 wherein R represents a vector, H and W are the number of pixels in the vertical and horizontal directions of the image respectively, and the foregoing steps can provide a numerically stable starting point for subsequent multi-scale feature encoding.

[0033] In step S3-1, multi-channel semantic feature extraction is performed on the input training sample image feature, and the spatial resolution is reduced;

[0034] Specifically, the shallow embedding starts multi-channel semantic extraction on the training sample image by using two layers of 3x3 convolution in cooperation with normalization, the step length of the first convolution is set to 2, which can not only align the color and texture, but also provide a smooth and noise-controlled base for the subsequent SGF-FL module; the spatial resolution in this stage is reduced to 1 / 2 of the input, and the output is F0∈R (H / 2)×(W / 2)×32 wherein R represents a vector, H and W are the number of pixels in the vertical and horizontal directions of the image respectively;

[0035] In step S3-2, in each encoding stage, the spectral branch, the global branch and the filter branch are developed in parallel by the three feature extraction branches of the SGF-FL module to obtain spectral information, global information and filter information in the input features;

[0036] The spectral information, the global information and the filter information are spliced in the channel dimension, added to the input residual after being compressed to the stage channel number by 1x1 convolution, and the stage representation is obtained to realize the space spectrum complementation and numerical stability;

[0037] In step S3-3, the input image features are down-sampled while extracting high-frequency and low-frequency information in combination with the wavelet change mode;

[0038] Specifically, to retain the edge and texture information required for reconstruction when down-sampling, the WPD module first performs two-dimensional Haar wavelet decomposition on the output of the SGF-FL module to obtain four components of LL, LH, HL and HH; then the three high-frequency components of LH, HL and HH are fused, and the pooling features of the input features are obtained by maximum pooling; then the fused high-frequency components and the pooling features are merged, and finally the down-sampling result which reduces the resolution and explicitly carries the high-frequency contribution is obtained by adding the light mapping and the LL low-frequency component through residual sum;

[0039] In step S3-4, steps S3-2 and S3-3 are repeated 2-3 times in multiple stages to deepen the semantics and gradually compress the spatial scale, while learning semantic features and continuously down-sampling the features;

[0040] In step S3-5, the SGF-FL module is used to extract the semantic features of deep features, and finally a multi-scale tensor set {F1∈R (H / 2)×(W / 2)×64 , F2∈R (H / 4)×(W / 4)×128 , F3∈R (H / 8)×(W / 8)×256} is obtained, where R represents a vector, H and W are the number of pixels in the vertical and horizontal directions of the image respectively.

[0041] The spectral branch in step S3-2 describes the local texture and detailed outline in the form of "3x3 standard convolution + 5x5 depth separable convolution + 1x1 channel mixing";

[0042] The global branch uses a windowed multi-head self-attention structure to establish long-distance dependence within the local window, and enhances the nonlinear representation through a feedforward network;

[0043] The filtering branch combines the Sobel convolution, which is fixed in the prior image processing, with a learnable bilateral filtering module. It uses the "dilation-erosion" differentiable approximation to achieve a closing operation, generating boundary buffer weights to selectively amplify true boundaries and suppress pseudo-high-frequency noise. The specific operation process is as follows: the input feature image is processed by Sobel convolution and a learnable bilateral filtering module to extract edge features and denoising features respectively. Then, the Tanh activation function is used to normalize the values ​​of the edge feature map. The "dilation-erosion" closing operation is then used to generate a boundary buffer for the boundary features. The boundary buffer is then multiplied with the input image features to achieve selective edge amplification. Finally, it is added to the denoising features. This process highlights edge features while improving the consistency of the ground features and suppressing the influence of noise through denoising.

[0044] The Sobel convolution is used to explicitly introduce classical image gradient operators to characterize directional edge information in the feature map. Its specific implementation is as follows: for the input feature map F∈R... H×W×C In terms of spatial dimensions, Sobel kernels K are used horizontally. x Sobel kernel K in the vertical direction y Perform convolution operations to obtain the corresponding gradient response G. x With G y ,in

[0045] ;

[0046] The edge magnitude map G is further calculated by performing a square root operation on a pixel-by-pixel basis.

[0047] ;

[0048] The Sobel convolution operator described above is predefined with fixed parameters and does not participate in backpropagation updates. Its role is to explicitly extract locations in the feature space where grayscale or feature response changes drastically, thereby obtaining direction-sensitive edge feature maps. The edge responses generated in this way provide reliable initial boundary cues for subsequent morphological closing operations, helping to reduce overfitting of the network to noisy gradients and improving the stability and interpretability of real-world object boundaries.

[0049] The bilateral filtering module is used to perform adaptive weighted smoothing of input features based on feature similarity. Its specific implementation process is carried out in three steps: similarity calculation, weight normalization, and weighted feature aggregation.

[0050] First, take the center feature F corresponding to any pixel position in the input feature map. input ∈R H×W×C For reference, it is adjusted to F∈R by scaling. C×H×W×1Then, the adjacent region feature N in the eight-neighborhood of the center feature is selected, the feature similarity is measured by calculating the mean absolute difference between the center feature and the adjacent feature, and an exponential decay function is introduced to construct the similarity score Aff, which is calculated as

[0051] ;

[0052] Wherein, Aff is the similarity score, Mean(.) represents the average operation on the feature channel dimension, and the exponential decay form can effectively suppress the adjacent pixels with large differences from the center feature, so that the feature similar region obtains higher response weight.

[0053] Subsequently, the similarity score Aff in the neighborhood range is normalized to obtain the bilateral filtering weight Weight, which is calculated as

[0054] ;

[0055] Wherein, Sum(.) represents the summation operation of all similarity scores in the neighborhood, and the normalization operation ensures that the sum of all neighborhood weights is 1, thereby avoiding unstable amplification of feature amplitude with neighborhood changes.

[0056] Finally, the neighborhood feature N is weighted and summed using the normalized bilateral filtering weight to obtain the output feature BF of the bilateral filtering module output , and the calculation process is

[0057] ;

[0058] Through the above calculation method, the bilateral filtering module can realize effective smoothing in the feature similar region, while adaptively reducing the cross-region feature propagation strength at the boundary position with large feature difference, thereby suppressing noise interference while maintaining the integrity of the feature boundary structure, providing stable and consistent noise reduction feature representation for the subsequent fusion with the Sobel convolution enhancement branch.

[0059] In step S4-1, the decoder performs shallow feature structure perception on the shallow feature F1 and the middle feature F2 to complete detail correction. The process first adjusts the spatial scale information of the shallow feature F1 to the same as the middle feature F2 through the CBAM module and the WPD module, then selectively amplifies the real edges and suppresses the pseudo edges and noise of the adjusted shallow feature F1 and the middle feature F2 by reusing the edge feature extraction method of the SGF-FL model in the encoding stage, and finally combines the two features through channel dimension combination to obtain the shallow-middle feature for subsequent feature fusion; wherein, the CBAM module is an attention module.

[0060] In step S4-2, for the middle layer feature F2 and the deep layer feature F3, the decoder adopts patch embedding with overlap, and then normalization and activation are performed to unify the 1x1 projection to the same channel dimension to obtain the middle-deep layer feature; the middle-deep layer feature takes into account the cross-stage semantic alignment and spatial detail compensation, so that the up-sampled features of each scale have fusibility at the common resolution of H / 4xW / 4, realizing cross-stage semantic alignment and context modeling;

[0061] In step S4-3, the shallow middle layer feature and the middle-deep layer feature are spliced in the channel dimension, and then re-calibrated by the structure perception module combined with channel attention and spatial attention to obtain the fusion feature; wherein the channel attention uses global average / max pooling and two layers of full connection to capture "which type of semantic is more important", and the spatial attention highlights "where is more important" through 7x7 convolution in the plane; finally, the fusion feature is regularized to the decoding channel dimension in the form of "3x3 convolution reshaping + 1x1 compression" to obtain a fusion representation with consistent structure, smooth texture and clear semantics.

[0062] In step S4-4, the fusion representation is gradually restored to the original resolution through two levels of bilinear up-sampling, and then avoids the checkerboard artifact through light convolution and normalization; a single layer of 1x1 convolution is used through the classification head to directly map to the numclass dimension, and a pixel-level probability map X output ∈R H×W×numclass wherein R represents a vector, H and W are the number of pixels in the longitudinal and transverse directions of the image respectively, and numclass refers to the number of classes output by the model.

[0063] The patch embedding with overlap in step S4-2 is specifically a linear projection in stride-overlap mode through 3x3 convolution.

[0064] In step S5, cross-entropy loss is used for supervision at the point annotation position.

[0065] Specifically, the cross-entropy loss is calculated only at the set of point annotations, so as to drive the end-to-end training under the condition of extremely low annotation ratio of 0.01% pixels; since the edge and frequency domain priori are introduced in the encoding stage, and the spatial consistency is constrained through the structure perception fusion in the decoding stage, the model can effectively suppress overfitting and maintain boundary sharpness under the condition of sparse supervision, and combined with the cosine annealing learning rate scheduling, the whole training process shows good convergence stability and cross-scene generalization ability; wherein the edge and frequency domain priori respectively refer to the edge feature extracted by the Sobel convolution in the SGF-FL module and the wavelet change down-sampling in the WPD module.

[0066] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0067] The present application can realize fine semantic segmentation of remote sensing images only by relying on point-level labeling, guaranteeing the basic learnability of the model under weak supervision, overcoming the problems of high cost and difficulty in scaling of traditional pixel-level labeling, and enabling comparable segmentation accuracy to full supervision methods under the premise of significantly improved labeling efficiency, thereby promoting the development and progress of large-scale remote sensing mapping technology.

[0068] Some additional features of the present application can be further described in the following description. Through examination of the following description and corresponding drawings, or understanding of the operation and training process of the embodiments, those skilled in the art can clearly realize some additional technical effects proposed by the present application. The features disclosed in the present application can be realized and obtained by practicing different modules, strategies and structures in specific embodiments, training processes and combination manners. BRIEF DESCRIPTION OF DRAWINGS

[0069] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute a limitation on the present application. In the drawings, the same reference numerals represent the same components. Among them,

[0070] Figure 1 It is a schematic diagram of the method framework of the present application.

[0071] Figure 2 It is a schematic diagram of the SGF-FL three-branch module structure of the present application.

[0072] Figure 3 It is a schematic diagram of the structure of the Sobel convolution and bilateral filtering module in the SGF-FL module of the present application.

[0073] Figure 4 It is a schematic diagram of the WPD module structure of the present application.

[0074] Figure 5 It is a schematic diagram of the HSA-Decoder structure of the present application. DETAILED DESCRIPTION

[0075] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts should be within the scope of protection of the present application.

[0076] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0077] Embodiment 1

[0078] As Figure 1 shown, a point annotation weakly supervised semantic segmentation method includes the following steps:

[0079] S1, model hyperparameter setting;

[0080] S1-1, basic channel construction and stage expansion;

[0081] A multi-stage encoder is constructed starting from the shallowest layer feature channel number C = 32, in order to gradually enhance the feature expression ability while controlling the amount of calculation; the spatial resolution of the feature map is reduced by 1 / 2 and the channel number is expanded by 2 times for each next stage, thereby forming a hierarchical pyramid of “resolution reduction, channel promotion”;

[0082] For example, when the input size is HxW, the output of Stage-1 is (H / 2)x(W / 2)x64, the output of Stage-2 is (H / 4)x(W / 4)x128, and if a deeper layer is enabled, the output can reach (H / 8)x(W / 8)x256. This design can maintain details in the shallow layer and strengthen semantics in the deep layer, so that the subsequent decoder can obtain both boundary and context information.

[0083] S1-2, using self-attention structure;

[0084] In order to establish long-distance dependency relationship at lower complexity, a windowed multi-head self-attention structure is used in the Transformer feature extraction layer of the multi-stage encoder; wherein 8 attention heads are fixedly set in each layer of the Transformer feature extraction layer, and the window side length is 8;

[0085] S1-3, training and inference setting;

[0086] In the training stage, the Adam optimizer is used, the initial learning rate is 0.001, the weight decay is 1e-4, and the cosine annealing strategy is used to gradually reduce the learning rate to obtain smooth convergence, and the cross-entropy loss is used as the model optimization loss calculation method; it is worth noting that only the pixels of the point annotation position calculate the loss, the loss function used by the model output and the point annotation label is the cross-entropy loss function; the unannotated pixels do not calculate the loss, so as to fit the extremely low labeling cost scene.

[0087] The cross-entropy loss function is a commonly used loss function in deep learning, which is prior art in the field, and calculates the loss between the model output and the sparse point annotation, which can be calculated by the skilled person without creative labor, and will not be described here.

[0088] S2, data preparation and normalization;

[0089] S2-1: remote sensing image cropping and data division;

[0090] In order to balance the memory usage and the integrity of the scene, the original remote sensing image is cropped into 256x256 patches, or when resources permit, the original remote sensing image is cropped into 512x512 patches. The patches are divided into training set, validation set and test set according to the ratio of 6:2:2. The patches in each data set are labeled by point annotation;

[0091] The point annotation is randomly sampled and labeled in the effective area corresponding to different categories, and the total labeling pixel ratio is 0.01%. The number of categories numclass is set for each data set. In order to reduce the training bias, attention should be paid to maintaining the balance between different categories during sampling;

[0092] S2-2: normalization and alignment;

[0093] The input patch is zero-mean and unit-variance normalized in the channel dimension to eliminate overall brightness and contrast drift caused by imaging conditions. If necessary, global brightness / contrast normalization or histogram matching can be further used to alleviate the domain difference of cross-scene and cross-time data. Finally, the training sample image X0∈R H×W×3 , where R represents a vector, H and W are the number of pixels in the vertical and horizontal directions of the image respectively, and the foregoing steps can provide a numerically stable starting point for subsequent multi-scale feature encoding.

[0094] S3, multi-scale feature encoding based on SGF-Encoder;

[0095] S3-1: shallow embedding;

[0096] The input training sample image feature is subjected to multi-channel semantic feature extraction and spatial resolution reduction.

[0097] Specifically, the shallow embedding starts multi-channel semantic extraction of the training sample image with two layers of 3x3 convolution combined with normalization. The step size of the first convolution is set to 2, which can not only align the color and texture, but also provide a smooth and noise-controlled base for the subsequent SGF-FL module. The spatial resolution at this stage becomes 1 / 2 of the input, F0∈R (H / 2)×(W / 2)×32wherein R represents a vector, H and W are the number of pixels in the vertical and horizontal directions of the image respectively;

[0098] S3-2: Three-branch joint feature learning is performed through the SGF-FL module;

[0099] As shown in the figure, at each encoding stage, the spectral branch, the global branch and the filter branch are developed in parallel through the three feature extraction branches of the SGF-FL module to obtain the spectral information, the global information and the filter information in the input features; Figure 2

[0100] The spectral information, the global information and the filter information are spliced in the channel dimension, and after being compressed back to the stage channel number through 1x1 convolution, they are added with the input residual to obtain the stage representation, realizing the space-spectrum complementation and numerical stability;

[0101] The spectral branch describes the local texture and detailed contour in the form of "3x3 standard convolution + 5x5 depth separable convolution + 1x1 channel mixing";

[0102] The global branch uses a windowed multi-head self-attention structure to establish long-distance dependence within the local window, and enhances the nonlinear representation through a feedforward network;

[0103] As shown in the figure, the filter branch combines the Sobel convolution in the image processing prior process with the learnable bilateral filtering module, and uses the "dilation-erosion" differentiable approximation to realize the closing operation, generates a boundary buffer weight to selectively amplify the real boundary and suppress the pseudo-high-frequency noise; the specific operation process is as follows: the input feature image is respectively extracted through Sobel convolution and learnable bilateral filtering module to extract edge features and noise reduction features, then the Tanh activation function is used to normalize the numerical value of the edge feature map, then the "dilation-erosion" closing operation is used to generate a boundary buffer area of the boundary feature, then the boundary buffer area is multiplied with the input image feature to selectively amplify the edge, and then the noise reduction feature is added to highlight the edge feature and improve the consistency of the ground object inside by noise reduction, thereby suppressing the influence of noise. Figure 3

[0104] The Sobel convolution is used to explicitly introduce a classical image gradient operator to describe the directional edge information in the feature map, and the specific implementation is as follows: the input feature map F H×W×C The horizontal direction Sobel kernel K x and the vertical direction Sobel kernel K y are respectively used for convolution operation to obtain the corresponding gradient responses G x and G y , wherein

[0105] ;​​

[0106] Further, the edge amplitude map G is calculated by a pixel-by-pixel square sum and square root operation, that is,

[0107] ;

[0108] The Sobel convolution operator described above is predefined and has fixed parameters, and does not participate in backpropagation update. Its role is to explicitly extract the position where the gray level or feature response changes dramatically in the feature space, thereby obtaining an edge feature map with direction sensitivity. The edge response generated in this way provides reliable initial boundary clues for subsequent morphological closing operation, which helps to reduce the overfitting of the network to noise gradients and improve the stability and interpretability of real object boundaries.

[0109] The bilateral filtering module is used for adaptive weighted smoothing of input features based on feature similarity. The specific implementation process is performed in three steps of similarity calculation, weight normalization and weighted feature aggregation.

[0110] First, taking the center feature F input ∈R H×W×C as the reference, it is adjusted to F∈R C×H×W×1 by scaling, and then the adjacent region feature N is selected in the eight-neighborhood range of the center feature. The feature similarity is measured by calculating the mean absolute difference between the center feature and the neighborhood feature, and an exponential decay function is introduced to construct the similarity score Aff, which is calculated as

[0111] ;

[0112] Where Mean(.) represents the average operation on the feature channel dimension, and the exponential decay form can effectively suppress the neighborhood pixels with large differences from the center feature, so that the feature similar region obtains higher response weight.

[0113] Subsequently, the similarity score Aff in the neighborhood range is normalized to obtain the bilateral filtering weight Weight, which is calculated as

[0114] ;

[0115] Where Sum(.) represents the summation operation of all similarity scores in the neighborhood, and the normalization operation ensures that the sum of all neighborhood weights is 1, thereby avoiding unstable amplification of feature amplitude with neighborhood changes.

[0116] Finally, the normalized bilateral filtering weight is used to weight and sum the neighborhood features N to obtain the output feature BF output of the bilateral filtering module, and the calculation process is

[0117] ;

[0118] Through the above calculation mode, the bilateral filtering module can realize effective smoothing in a feature similar region, while adaptively reducing the cross-region feature propagation strength at a boundary position with large feature difference, thereby suppressing noise interference while maintaining the integrity of the feature boundary structure, providing stable and consistent noise reduction feature representation for subsequent fusion with the Sobel convolution enhancement branch.

[0119] S3-3: Wavelet-preserving down-sampling is performed by the WPD module;

[0120] As shown in Figure 4 , the input image features are down-sampled while extracting high-frequency and low-frequency information in combination with the wavelet change mode;

[0121] Specifically, to preserve the edge and texture information required for reconstruction during down-sampling, the WPD module first performs two-dimensional Haar wavelet decomposition on the output of the SGF-FL module to obtain LL, LH, HL, and HH four components; then fuses the LH, HL, and HH three high-frequency components, and obtains the pooling features of the input features through maximum pooling; then merges the fused high-frequency components and the pooling features; finally, after light mapping, the LL low-frequency component is added through residual sum to form a down-sampled result that reduces the resolution and explicitly carries high-frequency contributions; this mechanism can significantly reduce the boundary blur and detail loss caused by only using pooling or step convolution, providing protection for fine reconstruction in the decoding stage.

[0122] S3-4: Multi-stage progressive coding;

[0123] Steps S3-2 and S3-3 are repeated 2-3 times in multiple stages to continuously deepen the semantics and gradually compress the spatial scale, learning semantic features while continuously down-sampling the features;

[0124] Taking an input of HxWx3 as an example, after Stage-1 and Stage-2, feature maps with spatial resolutions of (H / 2)x(W / 2)x64 and (H / 4)x(W / 4)x128 are obtained, respectively; if the scene is complex or requires stronger abstraction capability, a deep layer feature of (H / 8)x(W / 8)x256 can be obtained to enhance global consistency and class separability.

[0125] S3-5: Deep semantic extraction;

[0126] The SGF-FL module is used to extract the semantic features of deep features, and finally a multi-scale tensor set {F1∈R (H / 2)×(W / 2)×64 , F2∈R (H / 4)×(W / 4)×128 , F3∈R(H / 8)×(W / 8)×256}, where R denotes a vector, H and W are the number of pixels in the vertical and horizontal directions of the image, respectively.

[0127] S4, multi-level feature fusion decoding based on the HSA-Decoder as shown in Figure 5

[0128] S4-1: shallow edge enhancement and denoising

[0129] The decoder performs shallow feature structure perception on the shallow feature F1 and the middle feature F2 to complete detail correction. The process first adjusts the spatial scale information of the shallow feature F1 to the same as that of the middle feature F2 through a CBAM module and a WPD module, then selectively amplifies the true edges and suppresses the pseudo edges and noises of the adjusted shallow feature F1 and the middle feature F2 respectively by reusing the edge feature extraction mode of the SGF-FL model in the encoding stage, and finally combines the two groups of features by combining the channel dimensions to obtain shallow-middle features for subsequent feature fusion. The CBAM module is an attention module.

[0130] S4-2: overlapping patch embedding is adopted for the middle-deep features to realize cross-stage semantic alignment and context modeling.

[0131] The patch embedding with overlap is specifically a linear projection in a stride-overlap mode by using a 3x3 convolution.

[0132] For the middle feature F2 and the deep feature F3, the decoder adopts patch embedding with overlap, and then normalizes and activates to unify to the same channel dimension with 1x1 projection to obtain middle-deep features. The middle-deep features take into account the cross-stage semantic alignment and spatial detail compensation, so that the up-sampled features of each scale have fusibility at the common resolution of H / 4xW / 4, realizing cross-stage semantic alignment and context modeling.

[0133] S4-3, multi-scale splicing and structure perception fusion

[0134] In the unified scale, the shallow-middle features and the middle-deep features are spliced in the channel dimension, and then re-calibrated by a structure perception module combining channel attention and spatial attention to obtain fused features. The channel attention uses global average / max pooling and two layers of full connection to capture "which type of semantics is more important", and the spatial attention highlights "where is more critical" through a 7x7 convolution in the plane. Finally, the fused features are regularized to the decoding channel dimension in the form of "3x3 convolution reshaping + 1x1 compression" to obtain structure-consistent, texture-smooth and semantic-distinct fused representations.

[0135] S4-4: up-sampling reconstruction and classification output ​

[0136] The fusion characterizes a two-stage bilinear up-sampling step by step to recover to the original resolution, and then avoids the chessboard artifacts through light convolution and normalization; a single-layer 1x1 convolution is used through the classification head to directly map to the numclass dimension, and a pixel-level probability map X of the same size as the input is generated through Softmax output ∈R H×W×numclass wherein R represents a vector, H and W are the number of pixels in the longitudinal and transverse directions of the image respectively, and numclass refers to the number of classes output by the model.

[0137] The light classifier structure helps to concentrate the expression ability in the encoding and fusion links, while reducing the deployment delay.

[0138] S5, loss and training strategy setting.

[0139] The cross-entropy loss is used to supervise the point annotation position;

[0140] Specifically, the cross-entropy loss is calculated at the set of point annotations to drive the end-to-end training under the extremely low annotation ratio of 0.01% pixels, and due to the introduction of edge and frequency domain prior in the encoding stage and the constraint of spatial consistency through structure perception fusion in the decoding stage, the model can effectively suppress overfitting and maintain boundary clarity under sparse supervision conditions, combined with cosine annealing learning rate scheduling, the whole training process shows good convergence stability and cross-scene generalization ability; wherein the edge and frequency domain prior respectively refer to the edge features extracted by the Sobel convolution in the SGF-FL module and the wavelet change down-sampling in the WPD module.

[0141] It should be noted that all the features disclosed in the specification, or the steps in all the disclosed methods or processes, can be combined in any manner, except that mutually exclusive features and / or steps cannot be combined.

[0142] In addition, the above specific embodiments are exemplary, and those skilled in the art can think of various solutions under the inspiration of the disclosure of the present application, and these solutions also belong to the disclosed scope of the present application and fall within the protection scope of the present application. Those skilled in the art should understand that the specification and drawings of the present application are illustrative and do not constitute a limitation on the claims. The protection scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for point annotation weakly supervised semantic segmentation, characterized in that, Comprise the following steps: S1, model hyperparameter setting; S1-1: basic channel construction and stage expansion; Take the shallowest layer feature channel number C=32 as the starting point to construct a multi-stage encoder, and reduce the spatial resolution of the feature map by 1 / 2 and expand the channel number by 2 times every time entering the next stage; S1-2: adopt self-attention structure; The windowed multi-head self-attention structure is adopted in the Transformer feature extraction layer of the multi-stage encoder; S1-3: training and inference setting; In the training stage, the Adam optimizer is adopted, the initial learning rate is 0.001, the weight decay is 1e-4, and the learning rate is gradually reduced to obtain smooth convergence combined with the cosine annealing strategy, and the cross entropy loss is used as the model optimization loss calculation method; S2, data preparation and normalization; S2-1: remote sensing image patching and data division; The original remote sensing image is cut into 256x256 patches, or the original remote sensing image is cut into 512x512 patches, and the patches are divided into training set, validation set and test set according to the ratio of 6:2:2, and the patches in each data set are labeled by point annotation; S2-2: normalization and alignment; The input patch is normalized in the channel dimension with zero mean and unit variance, and further normalized with global brightness / contrast or histogram matching. Finally, the training sample image X0eR H×W×3 where R represents a vector, H and W are the number of pixels in the vertical and horizontal directions of the image, respectively, i.e., two dimensions of the three-dimensional vector. S3, multi-scale feature coding based on SGF-Encoder; S3-1: shallow embedding; The input training sample image features are subjected to multi-channel semantic feature extraction and spatial resolution reduction; S3-2: three-branch joint feature learning through SGF-FL module; In each coding stage, the spectral branch, global branch and filtering branch are developed in parallel through the three feature extraction branches of the SGF-FL module to obtain the spectral information, global information and filtering information in the input features; The spectral information, global information and filtering information are spliced in the channel dimension, and after being compressed back to the stage channel number by 1x1 convolution, they are added with the input residual to obtain the stage representation; S3-3: wavelet preserving down-sampling through WPD module; The input image features are down-sampled while extracting high-frequency and low-frequency information by combining the wavelet change mode; S3-4: multi-stage progressive coding; Steps S3-2 and S3-3 are repeated 2 or 3 times in multiple stages to continuously deepen the semantics and gradually compress the spatial scale, while learning semantic features and continuously downsampling features; S3-5: deep semantic refinement; The semantic features of deep features are extracted by the SGF-FL module, and finally a multi-scale tensor set {F1∈R (H / 2)×(W / 2)×64 , F2∈R (H / 4)×(W / 4)×128 , F3∈R (H / 8)×(W / 8)×256} is obtained, wherein F1 is a shallow feature in the semantic feature, F2 is a middle feature in the semantic feature, F3 is a deep feature in the semantic feature, R represents that the feature is a three-dimensional vector, H and W are respectively the number of pixels in the longitudinal direction and the transverse direction of the image; S4, multi-level feature fusion decoding based on HSA-Decoder; S4-1: shallow edge enhancement and denoising; The decoder performs shallow feature structure perception on the shallow feature F1 and the middle feature F2 to complete detail correction. First, the spatial scale information of the shallow feature F1 is adjusted to the same as the middle feature F2 through the CBAM module and the WPD module, then the edge feature extraction method of the SGF-FL model in the coding stage is reused to selectively amplify the real edges of the adjusted shallow feature F1 and the middle feature F2, and suppress the pseudo edges and noise, and finally the two groups of features are combined through channel dimension combination to obtain shallow middle layer features; S4-2: overlapping patch embedding is adopted for mid-deep features to realize cross-stage semantic alignment and context modeling; For mid-level features F2 and deep-level features F3, the decoder adopts patch embedding with overlap, and then normalization and activation are performed to unify the 1x1 projection to the same channel dimension to obtain mid-deep features; S4-3, multi-scale splicing and structure perception fusion; In the unified scale, the shallow and mid-level features are spliced in the channel dimension, and then the structure perception module of joint channel attention and spatial attention is used for re-calibration to obtain the fusion features; Finally, the fusion features are regularized into the decoding channel dimension in the form of 3x3 convolution shaping + 1x1 compression to obtain the fusion representation. S4-4: up-sampling reconstruction and classification output; The fusion feature is gradually restored to the original resolution through two-stage bilinear up-sampling, and then a light convolution and normalization are used to avoid chessboard artifacts; a single-layer 1x1 convolution is used through the classification head to directly map to the numclass dimension, and a pixel-level probability map X of the same size as the input is generated through Softmax output ∈R H×W×numclass wherein R represents a vector, H and W are the number of pixels in the longitudinal and transverse directions of the image respectively, and numclass refers to the number of classes output by the model; S5, loss and training strategy setting; The cross-entropy loss is used to supervise the point annotation position.

2. The point annotation weakly supervised semantic segmentation method according to claim 1, characterized in that, In step S1-2, 8 attention heads are fixedly arranged in each layer of the Transformer feature extraction layer, and the window side length is 8.

3. The point annotation weakly supervised semantic segmentation method according to claim 2, characterized in that, In step S2-1, the point annotation is randomly sampled and labeled in the effective area corresponding to different categories, and the total annotation pixel ratio is 0.01%, and the number of categories numclass is set for each data set.

4. The point annotation weakly supervised semantic segmentation method according to claim 3, characterized in that, The step S3-1 is specifically that shallow embedding is performed on the training sample image in a multi-channel semantic starting mode with two layers of 3*3 convolution cooperating with normalization, and the step length of the first convolution is set to 2; the spatial resolution in this stage becomes 1 / 2 of the input, and the output is F0∈R (H / 2)×(W / 2)×32 wherein R represents a vector, H and W represent the number of pixels in the longitudinal direction and the transverse direction of the image respectively. In step S3-3, the WPD module first performs two-dimensional Haar wavelet decomposition on the output of the SGF-FL module to obtain LL, LH, HL and HH four components; then the LH, HL and HH three high-frequency components are fused, and the pooling features of the input features are obtained through maximum pooling; then the fused high-frequency components and the pooling features are combined, and finally the light mapping is performed and the LL low-frequency component is added through residual sum to form the down-sampling result.

5. The point annotation weakly supervised semantic segmentation method according to claim 4, characterized in that, In step S3-2, the spectral branch is depicted by 3x3 standard convolution + 5x5 depth separable convolution + 1x1 channel mixing combination to depict local texture and detailed contour; The global branch adopts a windowed multi-head self-attention structure to establish long-distance dependence within the local window, and enhances the non-linear representation through a feedforward network; The filtering branch combines the shaped Sobel convolution in the image processing prior process with the learnable bilateral filtering module, and is supplemented by a micro-realizable closed operation of dilation-erosion to generate a boundary buffer weight; the specific operation process is as follows: the input feature image is extracted through Sobel convolution and learnable bilateral filtering module to obtain edge feature and noise reduction feature, then the value of the edge feature map is normalized by using the Tanh activation function, then the boundary buffer zone of the edge feature is generated through the closed operation of dilation-erosion, then the selective amplification of the edge is realized by multiplying the boundary buffer zone with the input image feature, and then the noise reduction feature is added.

6. The point annotation weakly supervised semantic segmentation method according to claim 5, characterized in that, In step S4-1, the CBAM module is an attention module.

7. The point annotation weakly supervised semantic segmentation method according to claim 6, characterized in that, In step S4-2, the overlapping patch embedding is specifically a linear projection in the stride-overlap mode by using 3x3 convolution.

8. The point annotation weakly supervised semantic segmentation method according to claim 7, characterized in that, In step S5, the cross-entropy loss is calculated only at the set of point annotations to drive the end-to-end training under the annotation ratio of 0.01% pixels.

Images