Generative adversarial frequency perception image segmentation network method

Abstract

The invention belongs to the technical field of remote sensing image segmentation, and particularly relates to a method for generating an adversarial frequency perception image segmentation network, which comprises the following steps of: 1, acquiring a public remote sensing image data set; 2, inputting the remote sensing image data set into a self-adaptive frequency driving feature enhancement module, and decoupling high-frequency features and low-frequency features; the frequency perception adversarial feature enhancement generates more real high and low frequency features through a generative adversarial network, and the resolution capability of the model is improved; and the CLIP text encoder converts the text description into semantic feature vectors to provide support for subsequent cross-modal matching. According to the remote sensing image semantic segmentation method, innovative design is carried out in the aspects of multi-modal fusion, high and low frequency information extraction, adversarial training and the like for remote sensing image semantic segmentation tasks with complex modals, and the segmentation precision, generalization ability and robustness are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image segmentation, and specifically to a method for generating an adversarial frequency-aware image segmentation network. Background Art

[0002] Remote sensing image segmentation is an important research direction in the cross-field of computer vision and remote sensing technology, aiming to perform pixel-level classification and regional division of different ground object categories in remote sensing images. With the rapid development of remote sensing technologies such as satellite imaging, unmanned aerial vehicles, and aerial photography, a large amount of high-resolution remote sensing data has been acquired. How to efficiently and accurately extract the ground object information of interest from remote sensing images has become the core issue of research.

[0003] The Chinese patent publication number is "CN119559200A", and the name is "A method for segmenting ultra-high-resolution remote sensing images based on frequency domain information fusion"; this network adopts a method for segmenting ultra-high-resolution remote sensing images based on frequency domain information fusion, and respectively extracts local detail features and global context features through a dual-branch encoder, effectively improving the parsing ability of fine-grained targets and strengthening the global information understanding ability at the same time. Through a multi-scale aggregation enhancement module, features of different scales are fused, improving the problem of incomplete consideration of large-scale structures and detail information. However, this method still has limitations in the extraction and optimization of high and low frequency features. The feature separation process may lead to the loss of detail information, thereby affecting the segmentation accuracy of the model for small sample categories. In addition, in a multi-category mixed scenario, this method lacks an effective semantic alignment mechanism, which may lead to confusion between different categories and affect the final segmentation effect. Therefore, how to accurately extract high and low frequency features, improve the adaptability of the model to small sample categories, and optimize the cross-category segmentation ability to enhance the generalization performance has become an important challenge that researchers in this field urgently need to solve. Summary of the Invention

[0004] (1) Technical Problems to be Solved

[0005] In view of the deficiencies of the prior art, the present invention provides a method for generating an adversarial frequency-aware image segmentation network, which solves the problems proposed in the above background art.

[0006] (2) Technical Solutions

[0007] The present invention specifically adopts the following technical solutions to achieve the above objectives:

[0008] A method for generating an adversarial frequency-aware image segmentation network includes the following steps:

[0009] Step 1, obtain a publicly available remote sensing image dataset;

[0010] Step 2: Input the remote sensing image dataset into the adaptive frequency-driven feature enhancement module to decouple high-frequency features and low-frequency features; frequency-aware adversarial feature enhancement generates more realistic high- and low-frequency features through a generative adversarial network to improve the model's resolution ability; the CLIP text encoder converts the text description into a semantic feature vector to support subsequent cross-modal matching.

[0011] Step 3: Input the input image and text features into the semantic-guided feature selection module to perform cross-modal alignment and cross-modal similarity calculation of the image and text features, and filter out the image features that match the target semantics.

[0012] Step 4: Input the optimized image features and the spatial adaptation weight into the dynamic visual-semantic fusion module to adaptively adjust the matching degree of visual and semantic features. Through the semantic enhancement weight allocation and residual learning mechanism, ensure that multi-modal information is fully fused in the shared feature space to improve the segmentation accuracy.

[0013] Step 5: Input the dynamically optimized visual-semantic fusion features into the multi-level semantic detail restoration network model to restore the pixel-level segmentation details and generate the final remote sensing image segmentation mask.

[0014] Further, in Step 2, the remote sensing image dataset is input into the adaptive frequency-driven feature enhancement module to obtain high-frequency and low-frequency features. The adaptive frequency-driven feature enhancement module includes a frequency-aware information fusion module, a pooling layer, a bilinear interpolation, a frequency-aware adversarial feature enhancement module, a convolutional layer, and a fully connected layer.

[0015] Input the remote sensing image dataset into the adaptive frequency-driven feature enhancement module, and use octave convolution to decouple the image features into high-frequency features and low-frequency features; then use the frequency-aware adversarial feature enhancement module to optimize the high- and low-frequency features.

[0016] Further, in Step 3, input the optimized image features and text features. Map the optimized image features and text features to the same dimension through linear mapping respectively, and then calculate the correlation weights between the two sets of features through the cross-modal attention mechanism to generate a weighted fusion vector; finally, input the vector into a multi-layer fully connected network for deep integration to obtain the final fusion information.

[0017] Further, in Step 4, after the spatial adaptation weight is normalized and maximized by Softmax, then the processed weight mapping is deeply fused with the original image features, and then the final fusion features are obtained through convolution and fully connected layers.

[0018] Furthermore, in step 5, a deep convolution and upsampling strategy is adopted to gradually restore pixel-level segmentation details, and a multi-task loss function is introduced to constrain and optimize the prediction results at different levels, and finally a high-quality remote sensing image segmentation result is output.

[0019] Furthermore, in step 5, a multi-task loss function is used to train the multi-level semantic detail restoration network model:

[0020] The multi-task loss function includes semantic segmentation loss, generative adversarial loss, frequency consistency loss, and cross-modal alignment loss;

[0021] The semantic segmentation loss is a loss that combines cross-entropy loss and Dice Loss, which is used to comprehensively optimize large and small targets, ensure that the finally generated segmentation mask matches the true label, and improve the pixel-level classification accuracy. Its calculation formula is:

[0022] L seg = λ1L CE + λ2L dice

[0023]

[0024] where λ1 and λ2 are hyperparameters, N is the total number of pixels in the image, y i is the true pixel class label, and p i is the class probability predicted by the model;

[0025] The generative adversarial loss makes the generated high- and low-frequency features more real and improves the generalization ability. Its calculation formula is:

[0026]

[0027] where L GAN represents the loss function of the generator, and L D represents the loss function of the discriminator, F H , F L is the true high- and low-frequency feature, is the generated high- and low-frequency feature;

[0028] The frequency consistency loss is used to limit the generation error of high- and low-frequency features and ensure that the features generated by GAN can effectively enhance the segmentation ability. Its calculation formula is:

[0029]

[0030] where ‖‖ represents the mean square error;

[0031] The cross-modal alignment loss is used to let the text description guide image segmentation, improve semantic consistency, and enhance the generalization ability of the model for few-shot classes. Its calculation formula is:

[0032]

[0033] F agg is an image feature, and F H is a text feature;

[0034] The total loss is composed of the losses of the above four parts, combining segmentation accuracy, adversarial optimization, frequency consistency, and cross-modal alignment. The total loss L total has the following specific calculation formula:

[0035] L total = λ1L seg + λ2L GAN + λ3L freg + λ4L cm

[0036] Among them, λ1, λ2, λ3, and λ4 are hyperparameters of the loss function, used to control the influence of different losses.

[0037] (III) Beneficial Effects

[0038] Compared with the prior art, the present invention provides a method for generating an adversarial frequency-aware image segmentation network, having the following beneficial effects:

[0039] 1. The present invention proposes an adaptive frequency-driven feature enhancement module, which optimizes the feature expression by using the generation and adversarial training of high-frequency and low-frequency features, making the features of the segmentation input more real and improving the model's ability to capture details of complex ground objects.

[0040] 2. The present invention proposes a semantic-guided feature selection module, which guides image segmentation through text information, enhances the model's semantic understanding ability, ensures that the segmentation task is consistent with the text description, thereby effectively preventing the model from relying only on visual features, improving the ability to distinguish different categories, and is particularly suitable for remote sensing scenarios with high inter-class similarity and large intra-class differences.

[0041] 3. The present invention proposes a dynamic visual-semantic fusion module, which can adaptively adjust the weight distribution of visual and semantic features, ensure the effective fusion of information in different modalities, prevent a single modality from dominating the segmentation result, and thus improve the adaptability and generalization ability of the model. At the same time, the present invention proposes a multi-task loss function, including semantic segmentation loss, generative adversarial loss, frequency consistency loss, and cross-modal alignment loss. By optimizing multiple tasks end-to-end, the segmentation stability and robustness of the model are improved, ensuring the synergistic effect of information in different modalities and achieving accurate remote sensing image segmentation. Description of the Drawings

[0042] Figure 1Flow chart of a generative adversarial frequency-aware image segmentation network according to the present invention;

[0043] Figure 2 Network block diagram of a generative adversarial frequency-aware image segmentation network according to the present invention;

[0044] Figure 3 Structural diagram of the adaptive frequency-driven feature enhancement module according to the present invention;

[0045] Figure 4 Structural diagram of the frequency-aware information fusion module according to the present invention;

[0046] Figure 5 Structural diagram of the frequency-aware adversarial feature enhancement module according to the present invention;

[0047] Figure 6 Structural diagram of the semantic-guided feature selection module according to the present invention;

[0048] Figure 7 Structural diagram of the dynamic visual semantic fusion module according to the present invention;

[0049] Figure 8 Structural diagram of the multi-level semantic detail restoration network according to the present invention;

[0050] Figure 9 Schematic diagram comparing the corresponding evaluation indexes of the method proposed by the present invention with the prior art indexes. Detailed implementation manners

[0051] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0052] Embodiment

[0053] As Figures 1-9 shown, a generative adversarial frequency-aware image segmentation network method proposed in an embodiment of the present invention includes the following steps:

[0054] Step 1, obtain a publicly available remote sensing image dataset.

[0055] It should be noted that two datasets, namely AID Dataset and BigEarthNet Dataset, are selected for the remote sensing image dataset. AID covers multiple categories and complex scenarios, which helps to evaluate the segmentation performance of the model in diverse environments and verify the effectiveness of high-low frequency feature optimization and adversarial training. BigEarthNet provides multi-label classification capabilities and is suitable for cross-modal learning. Especially in image-text semantic matching, it can enhance the model's understanding and generalization ability of different categories.

[0056] Step 2: Input the remote sensing image dataset into the frequency-aware information fusion module to analyze different frequency components of the remote sensing image; and perform feature reconstruction. Then, input the obtained high-frequency and low-frequency features into the frequency-aware adversarial feature enhancement module to improve the authenticity of the high-low frequency features through a generative adversarial network.

[0057] It should be noted that as Figure 2 shown, input the remote sensing dataset into the network block diagram of the generative adversarial frequency-aware image segmentation network. As Figure 3 shown, the adaptive frequency-driven feature enhancement module consists of a frequency-aware information fusion module, a pooling layer, bilinear interpolation, a frequency-aware adversarial feature enhancement module, a 1×1 convolution, and a fully connected layer.

[0058] It should be further noted that the frequency-aware information fusion module, as Figure 4 shown, is used to decouple the high-frequency and low-frequency features of the remote sensing image and improve the detailed segmentation ability of the remote sensing image. First, perform high-low frequency decomposition on the input remote sensing image through octave convolution. The decomposed high-frequency features are enhanced through 1×1 convolution, and the low-frequency features are enhanced through 3×3 convolution. Then, they are respectively input into the residual blocks to maintain information consistency. At the same time, skip connections effectively alleviate the problems of gradient disappearance and gradient explosion.

[0059] It should be further noted that the frequency-aware adversarial feature enhancement module, as Figure 5 shown, improves the authenticity of the high-low frequency features through a generative adversarial network, making them closer to the distribution of real remote sensing data. First, perform an upsampling operation on the low-frequency features and then fuse them with the high-frequency features to obtain enhanced high-frequency features. At the same time, perform a pooling operation on the high-frequency features and then fuse them with the low-frequency features to obtain enhanced low-frequency features.

[0060] First, the enhanced low-frequency features are passed through residual blocks and skip connections to enhance the expressive ability of the low-frequency features and prevent information loss. Subsequently, the attention mechanism improves the attention to key regions. The transposed convolution layer performs spatial expansion of the features, enabling the low-frequency features to have better adaptability at multiple scales. Meanwhile, the enhanced high-frequency features are input into the self-attention mechanism to capture the spatial relationships of the high-frequency features and highlight key texture and edge information. Subsequently, depthwise separable convolutions are used to extract features, improving the computational efficiency while reducing the number of parameters.

[0061] The high-frequency features output from the frequency-aware feature enhancer are passed through a 3×3 convolutional layer to capture small-scale edge information, and then through a fully connected layer to map the features to the final discriminative dimension. Subsequently, the Sigmoid function is used to determine whether the high-frequency features are real. Similarly, for the low-frequency features output from the frequency-aware feature enhancer, first, a 7×7 convolutional layer is used to extract large-scale global information to improve the modeling ability of the low-frequency features. Subsequently, the fully connected layer maps to the final discrimination, and the Sigmoid function determines whether the low-frequency features are real. Finally, the multi-scale frequency adversarial discriminator outputs the authenticity scores for the high- and low-frequency features. The multi-scale frequency adversarial discriminator is trained using LD to more accurately distinguish real and generated features. The frequency-aware feature enhancer is trained using LGAN to make the generated features closer to real features. Through the mutual game between the multi-scale frequency adversarial discriminator and the frequency-aware feature enhancer, the generated features are continuously optimized to improve the segmentation effect.

[0062] The original text descriptions in the dataset cannot be directly input into the model. Therefore, it is necessary to first perform natural language reconstruction on the keywords and convert them into complete descriptive sentences. Subsequently, this text information is processed through the CLIP text encoder to generate high-dimensional semantic features for cross-modal alignment. This process not only enhances the model's ability to understand category semantic relationships but also promotes the deep fusion of visual and language information, thus effectively improving the segmentation performance of remote sensing images.

[0063] In step 3, the semantic-guided feature selection module first linearly maps the image features and text features, and then through the image-text reasoning network, filters out the image regions most relevant to the target semantics for cross-modal similarity calculation, calculates the similarity score between the image and text features, obtains the weighted information, weights the image features, and generates the semantically enhanced image features.

[0064] It should be noted that as Figure 6 shown, the semantic-guided feature selection module consists of a linear mapping, an image-text reasoning module, cross-modal similarity calculation, a softmax function, a convolutional layer, a residual block, a pooling layer, bilinear interpolation, and a fully connected layer.

[0065] In the semantic-guided feature selection module, first, the image features and text features of the input image are linearly mapped to obtain semantically enhanced image features and visually enhanced text features, projecting features of different modalities into the same feature space so that the image and text can be matched at the same scale. Meanwhile, query Q, key K, and value V are obtained. The image-text reasoning module uses a cross-modal attention mechanism to calculate and infer the attention weight A. Subsequently, the attention weight A and value V are weighted to obtain relatively strongly correlated image features FR. This network enhances the relatively strongly correlated image features and suppresses irrelevant features. The cross-modal similarity calculation performs dot-product similarity calculation between FR and the visually enhanced text features. Image features with high matching degrees are retained, and features with low matching degrees are suppressed. Subsequently, through Softmax normalization, the cross-modal attention weight matrix ACS is obtained, ensuring reasonable attention distribution for different categories. Then, FR and ACS are multiplied element by element, and through 1×1, 3×3 convolutions, and residual blocks, the training stability of the deep network is improved. The pooling layer and bilinear interpolation restore the spatial resolution of the features. Finally, the filtered features are integrated through a fully connected layer, and the final semantic information FO is generated. Through this series of steps, the image features are made more semantically matched with the text description, improving the segmentation accuracy.

[0066] Step 4: Input the optimized image features and spatial adaptation weights into the dynamic visual-semantic fusion module to adaptively adjust the matching degree between visual and semantic features. Through the semantic enhancement weight distribution and residual learning mechanism, ensure the full fusion of multi-modal information in the shared feature space and improve the segmentation accuracy.

[0067] It should be noted that, as Figure 7 shown, the cross-modal attention weight matrix ACS is passed through the Softmax function and maximization operation to obtain Fmax, retaining key visual detail information and avoiding excessive bias towards semantic guidance. The dot-product operation is performed on Fmax and Fo, enabling the semantic features to truly act on the visual features and forming the final fused features. Subsequently, 1×1 and 3×3 convolutions enhance local and global information. At the same time, residual blocks are used to solve the problem of gradient disappearance. Finally, the optimized visual-semantic fusion features are output through a fully connected layer. Through this series of steps, cross-modal optimization is achieved, but its main goal is to optimize semantic consistency.

[0068] Step 5: First, obtain pixel-level segmentation predictions through the trained network. Then, use a multi-task loss function to jointly optimize the network. Finally, use the trained segmenter to perform refined segmentation on the remote sensing image.

[0069] It should be noted that, as Figure 8As shown, the output features of the automatic dynamic vision-semantic fusion module are first used to obtain multi-scale context information through pooling operations of different scales (1×1, 3×3, 5×5 pooling) to capture information on large regions and local details in the image. Subsequently, the pooled image features are input into 3×3 convolutions and residual blocks to strengthen local detail information and ensure the accuracy of edge and detail restoration. The upsampling operation is used to ensure that the output of the detail restoration network can match the size of the original image. Finally, the segmentation head generates the final segmentation mask, with each pixel corresponding to a class label.

[0070] Subsequently, it should be noted that the multi-task loss function is used to train the multi-level semantic detail restoration network model, including semantic segmentation loss, generative adversarial loss, frequency consistency loss, and cross-modal alignment loss.

[0071] The semantic segmentation loss is a loss that combines cross-entropy loss and Dice Loss, used to comprehensively optimize large and small targets, ensure that the finally generated segmentation mask matches the true label, and improve the pixel-level classification accuracy. Its calculation formula is:

[0072] L seg = λ1L CE + λ2L dice

[0073]

[0074] where λ1 and λ2 are hyperparameters, N is the total number of pixels in the image, y i is the true pixel class label, and p i is the class probability predicted by the model.

[0075] The generative adversarial loss makes the generated high- and low-frequency features more realistic and improves the generalization ability. Its calculation formula is:

[0076]

[0077] where L GAN represents the loss function of the generator, L D represents the loss function of the discriminator, F H , F L is the true high- and low-frequency feature, is the generated high- and low-frequency feature.

[0078] The frequency consistency loss is used to limit the generation error of high- and low-frequency features and ensure that the features generated by the GAN can effectively enhance the segmentation ability. Its calculation formula is:

[0079]

[0080] where ‖‖ represents the mean square error.

[0081] The cross-modal alignment loss is used to let the text description guide image segmentation, improve semantic consistency, and enhance the generalization ability of the model for few-shot categories. Its calculation formula is as follows:

[0082]

[0083] F agg is the image feature, and F H is the text feature.

[0084] The total loss is composed of the losses of the above four parts, combining segmentation accuracy, adversarial optimization, frequency consistency, and cross-modal alignment. The specific calculation formula of the total loss L total is as follows:

[0085] L total = λ1L seg + λ2L GAN + λ3L freg + λ4L cm

[0086] Among them, λ1, λ2, λ3, and λ4 are hyperparameters of the loss function, used to control the influence of different losses.

[0087] Comparing the corresponding evaluation indexes of the method proposed by the present invention with the existing technology indexes, as Figure 9 shown, it can be seen from the figure that the accuracy of the method proposed by the present invention is significantly higher than that of the existing technology, indicating the effectiveness of this method.

[0088] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for generating an adversarial frequency-aware image segmentation network, characterized in that: It includes the following steps: Step 1, obtain a publicly available remote sensing image dataset; Step 2, input the remote sensing image dataset into an adaptive frequency-driven feature enhancement module to decouple high-frequency features and low-frequency features; Frequency-aware adversarial feature enhancement generates more realistic high- and low-frequency features through a generative adversarial network to improve the resolution ability of the model; the CLIP text encoder converts text descriptions into semantic feature vectors to support subsequent cross-modal matching; Step 3, input the input image and text features into a semantic-guided feature selection module to perform cross-modal alignment and cross-modal similarity calculation of the image and text features, and filter out the image features that match the target semantics; Step 4, input the optimized image features and spatial adaptation weights into a dynamic visual-semantic fusion module to adaptively adjust the matching degree of visual and semantic features. Through semantic enhancement weight allocation and residual learning mechanisms, ensure that multi-modal information is fully fused in the shared feature space to improve the segmentation accuracy; Step 5, input the visually-semantically fused features optimized dynamically into a multi-level semantic detail restoration network model to restore pixel-level segmentation details and generate the final remote sensing image segmentation mask.

2. The method for generating an adversarial frequency-aware image segmentation network according to claim 1, wherein: In step 2, the remote sensing image dataset is input into the adaptive frequency-driven feature enhancement module to obtain high-frequency and low-frequency features. The adaptive frequency-driven feature enhancement module includes a frequency-aware information fusion module, a pooling layer, a bilinear interpolation, a frequency-aware adversarial feature enhancement module, a convolutional layer, and a fully connected layer; Input the remote sensing image dataset into the adaptive frequency-driven feature enhancement module, and use octave convolution to decouple the image features into high-frequency features and low-frequency features; Then, use the frequency-aware adversarial feature enhancement module to optimize the high- and low-frequency features.

3. The method for generating an adversarial frequency-aware image segmentation network according to claim 1, wherein: In step 3, input the optimized image features and text features. Map the optimized image features and text features to the same dimension through linear mapping respectively. Subsequently, calculate the correlation weights between the two sets of features through a cross-modal attention mechanism to generate a weighted fusion vector; finally, input this vector into a multi-layer fully connected network for deep integration to obtain the final fusion information.

4. A method for generating an adversarial frequency-aware image segmentation network according to claim 1, characterized in that: In step 4, after the spatial adaptation weights are normalized and maximized by Softmax, then map the processed weights and the original image features for deep fusion, and then obtain the final fusion features through convolutional and fully connected layers.

5. A method for generating an adversarial frequency-aware image segmentation network according to claim 1, characterized in that: In step 5, adopt a deep convolution and upsampling strategy to gradually restore pixel-level segmentation details, introduce a multi-task loss function to constrain and optimize the prediction results at different levels, and finally output a high-quality remote sensing image segmentation result.

6. A method for generating an adversarial frequency-aware image segmentation network according to claim 1, characterized in that: In step 5, a multi-task loss function is used to train the multi-level semantic detail restoration network model: The multi-task loss function includes a semantic segmentation loss, a generative adversarial loss, a frequency consistency loss, and a cross-modal alignment loss; The semantic segmentation loss is a loss that combines cross-entropy loss and Dice Loss, which is used to comprehensively optimize large and small targets, ensure that the finally generated segmentation mask matches the true label, and improve the pixel-level classification accuracy. Its calculation formula is: L seg = λ1L CE + λ2L dice where λ1 and λ2 are hyperparameters, N is the total number of pixels in the image, y i is the true pixel class label, and p i is the class probability predicted by the model; The generative adversarial loss makes the generated high- and low-frequency features more realistic and improves the generalization ability. Its calculation formula is as follows: Among them, L GAN represents the loss function of the generator, and L D represents the loss function of the discriminator. F H , and F L is the real high and low frequency features, and [generated high and low frequency features; The frequency consistency loss is used to limit the generation error of high- and low-frequency features and ensure that the features generated by the GAN can effectively enhance the segmentation ability. Its calculation formula is as follows: where ‖‖ represents the mean square error; The cross-modal alignment loss is used to let the text description guide image segmentation, improve semantic consistency, and enhance the model's generalization ability for few-shot categories. Its calculation formula is as follows: F agg is an image feature, F H is a text feature; The total loss is composed of the losses of the above four parts, combining segmentation accuracy, adversarial optimization, frequency consistency, and cross-modal alignment. The specific calculation formula of the total loss L total is as follows: L total = λ1L seg + λ2L GAN + λ3L freg + λ4L cm where λ1, λ2, λ3, λ4 are hyperparameters of the loss function, used to control the influence of different losses.

Images