Text knowledge guided context feature fusion network for ultrahigh-resolution satellite image

Abstract

The invention discloses a text knowledge guided context feature fusion network for an ultrahigh-resolution satellite image. Comprising an image encoder for receiving a scale adjustment image which is freely cut into a plurality of different scales in a multi-scale manner, a feature fusion module for performing staged fusion on image features, and an image decoder for reconstructing the image features and calculating cross entropy loss Lce, the adaptive text generation module is used for receiving a multi-scale label image and adaptively generating a category text for the multi-scale label image; the VLM text encoding module is used for generating a text feature vector from the text; the convolution module is used for performing feature conversion on features output by an image encoder; the global average pooling module is used for carrying out average pooling processing on output of the convolution module, the image-text comparison module is used for generating an image-text matrix and calculating similarity comparison entropy loss Litc, and the network can integrate context information of different stages and can extract a small number of categories of knowledge from a pre-training model so as to integrate cross-scale features.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing based on artificial intelligence, and in particular relates to a text knowledge-guided context feature fusion network for ultra-high resolution satellite images. Background Art

[0002] As more and more satellites are launched into orbit, massive amounts of remote sensing data are generated every day. With the development of technology, the resolution of these data will become higher and higher. The application of these massive data must be based on the ability to effectively interpret these data, otherwise it will affect the application of these data. For example, in the field of ultra-high resolution (UHR) satellite image segmentation, current methods are limited by the inherent class imbalance problem in satellite images, resulting in insufficient learning of specific class features, especially minority class features, which affects the training of artificial intelligence on minority classes in remote sensing satellite images, thereby affecting the deeper resolution of remote sensing satellite images. Summary of the invention

[0003] In order to solve the above problems existing in the prior art, the present invention proposes a text knowledge guided context feature fusion network for ultra-high resolution satellite images.

[0004] A text knowledge guided context feature fusion network for ultra-high resolution satellite images, wherein the ultra-high resolution satellite images are freely multi-scale cropped into multiple scale-adjusted images of different scales, and the scale-adjusted images are semantically segmented and annotated to form label images, characterized in that the network comprises:

[0005] An image encoder, comprising a plurality of stage encoders adapted to the scale-adjusted image in number, and the image features of the different scale-adjusted images are extracted by the corresponding plurality of stage encoders in a sequence of gradually increasing depth and gradually decreasing area in a plurality of stages, and each stage encoder obtains a plurality of feature images of the corresponding stage;

[0006] The feature fusion module includes a plurality of sub-feature fusion modules whose number matches that of the stage encoder and respectively correspond to one stage encoder, each sub-feature fusion module receives a plurality of feature images provided by the encoder of the corresponding stage, and fuses a plurality of features to obtain a feature fusion image of the corresponding stage, and the feature fusion images of the corresponding stages respectively obtained by the plurality of sub-feature fusion modules form a feature fusion image set;

[0007] The image decoder accepts the feature fusion image set, reconstructs the multi-image features, generates a reconstructed image, and calculates the cross entropy loss Lce generated by the pixel-by-pixel classification training of the input image;

[0008] The adaptive text generation module accepts the label image and automatically generates the category text of the corresponding image for the full-width category area not less than the set width ratio according to the semantic segmentation of the label image. The image is divided into regions for the category area not less than the set ratio. The category text of the corresponding region is automatically generated for each region with a category area not less than the set domain ratio. If any category area of ​​the three regions is less than the set domain ratio, a more fine-grained regional division is performed.

[0009] The VLM text encoding module encodes the text information in the adaptive text generation module and generates a text feature vector T with a dimension of 1×d;

[0010] The convolution module performs convolution processing on the feature image output by the encoder in the final stage to generate the converted image features;

[0011] A global average pooling module: performing average pooling processing on a transformed feature output by the convolution module to generate an image feature vector I of the same dimension 1×d as the transformed feature;

[0012] The image-text comparison module generates an image-text matrix based on the text feature vector and the image feature vector, and calculates the similarity comparison entropy loss Litc used to generate the similarity score for image-text conversion.

[0013] Furthermore, the multiple rescaled images are annotated with semantic segmentation, one of which maintains the original ground sampling distance and the others provide contextual information.

[0014] Furthermore, the multiple sub-feature fusion modules are divided into two categories: a low-level sub-feature fusion module for fusing underlying image features and a high-level sub-feature fusion module for fusing advanced image features. When the image features are extracted in sequence by multiple stages, the first half of the multiple stages uses the low-level sub-feature fusion module to fuse the extracted image features, and the second half uses the high-level sub-feature fusion module to fuse the extracted image features.

[0015] Furthermore, the low-level sub-feature fusion module is based on a convolutional structure and adopts the following convolution mechanism expression:

[0016]

[0017] Where: F fusion represents the image fusion equation;

[0018] Conv1×1(·) and Conv5×5 represent 1×1 and 5×5 convolution calculations;

[0019] Concate(·,·,·,·,) means concatenating four image features;

[0020] Fc1, Fc2, Fc3, and Fc4 represent four new feature maps generated by concatenation, where c represents the category and C is an integer. Here c1 to c4 refer to four categories.

[0021] Furthermore, the high-level sub-feature fusion module is based on the attention mechanism and adopts the following expression:

[0022] K = Concate(Flatten(F c2 ),Flatten(F c3 ),Flatten(F c4 ))

[0023] V = Concate(Flatten(F c2 ),Flatten(F c3 ),Flatten(F c4 ))

[0024] Q = Flatten (F c1 )

[0025] Where: K represents the key matrix, Q represents the query matrix, V represents the value matrix, Flatten(·) represents the operation of concatenating the obtained 2D feature images into a one-dimensional vector, and Concate(·,·,·,) represents the operation of concatenating the 3 feature maps in the brackets.

[0026] Furthermore, the cross entropy loss Lce generated by the pixel-by-pixel classification training is expressed as follows:

[0027]

[0028] Where: N represents the number of pixels in the image, and C represents the total number of categories. For each pixel i, represents the true label using one-hot encoding, that is, if pixel i belongs to category c, then otherwise refers to the predicted probability that pixel i belongs to category c.

[0029] Furthermore, the image-text conversion similarity score is the similarity score between image and text and text and image after calculating the softmax normalization, using the following expression:

[0030]

[0031] Where s(·,·) represents the cosine similarity between two vectors, and p i2t (I) is the given text T m The similarity score of image I when p t2i(T) is a given image I m is the recognition score of text T at that time. In practice, the mean is calculated. I refers to image features, T refers to text features, τ is the temperature coefficient, and M represents the total number of texts.

[0032] 1. Furthermore, the similarity comparison entropy loss Litc is expressed as follows:

[0033]

[0034] In the formula, represents the cross entropy function, represents the expected value, y i2t Represents the true label of the text, i2t refers to the i-th pixel to the m-th text, and t2i refers to the distance from the m-th text to the i-th pixel.

[0035] Compared with the prior art, the above solution has the following beneficial effects:

[0036] The present invention proposes a text knowledge guided context feature fusion network, which is provided with a feature fusion module and an image-text comparison module. The feature fusion module combines the convolution mechanism with the attention mechanism to integrate features and context information at different stages; the image-text comparison module integrates a pre-trained visual language model into a semantic segmentation framework, and utilizes these pre-trained models to enable richer and more robust knowledge to guide the learning of the segmentation model, thereby significantly improving the model performance. The knowledge of a few categories can be extracted from the pre-trained model to integrate cross-scale features, so that the image backbone network integrates the diversified information provided by the pre-trained model, thereby surpassing the general semantic segmentation model and the existing open source model in image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the network architecture diagram of the present invention;

[0038] Figure 2 Schematic diagram of the fusion process of the feature fusion module;

[0039] Figure 3 Implementation effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0041] Embodiment 1:

[0042] like Figure 1 As shown, the ultra-high resolution satellite image 1 is freely multi-scale cropped into four mutually inclusive scaled images. For example, if the original image is 10000*10000 pixels, one can first select 1024*1024, 512*512, 256*256, 128*128, and four scaled images. The four scaled images are semantically segmented and annotated to form four corresponding label images 3 with text information, one of which maintains the original ground sampling distance, and the others provide context information.

[0043] Based on such image information, the text knowledge-guided context feature fusion network for ultra-high resolution satellite images described in this embodiment includes: an image encoder, a feature fusion module, an image decoder, an adaptive text generation module, a VLM text encoding module, a convolution module, a global average pooling module, and an image-text comparison module.

[0044] The above-mentioned image encoder includes four stage encoders for image adaptation with quantity and scale adjustment, denoted as stage1, stage2, stage3, and stage4. Four stages are used to extract image features in the label image, and the image features are extracted in stages in the order of gradually deepening the depth and gradually decreasing the area. Each stage encoder obtains four feature images of the corresponding stage, and each feature image includes different image features such as: boundaries, textures, etc.

[0045] Furthermore, the image encoder of this embodiment adopts the visual-language model GeoRSCLIP specially used in the remote sensing field to provide auxiliary knowledge to improve the feature extraction quality of the image encoder.

[0046] The feature fusion module includes four sub-feature fusion modules whose number matches that of the stage encoder and respectively corresponds to one stage encoder. Each sub-feature fusion module receives four feature images provided by the encoder of the corresponding stage, and fuses the features extracted from each feature image to obtain a feature fusion image of the corresponding stage. The feature fusion images of the corresponding stages obtained by the multiple sub-feature fusion modules respectively form a feature fusion image set.

[0047] The above-mentioned image decoder receives a feature fusion image set, reconstructs multiple image features to generate a reconstructed image, and generates a cross entropy loss Lce for pixel-by-pixel classification training of the input image. The cross entropy loss Lce is one of the network loss functions of the present invention, and another one will be mentioned below.

[0048] The above-mentioned adaptive text generation module accepts the label image, and based on the semantic segmentation of the label image, automatically generates the category text of the corresponding image for the full-width category area not less than the set width ratio, divides the image into regions for the category area not less than the set ratio, and automatically generates the category text of the corresponding region for each region category area not less than the set domain ratio. If the proportion of the main category in any of the three regions is less than 50%, the image will continue to be segmented into finer granularity.

[0049] For example, for a full-frame image, the image ratio is set to 80%. If the image features of the main category in the full-frame image account for no less than 80% of the entire image ratio, this category is used to describe the image, and the generated text is a satellite image of this category. Otherwise, the image is evenly divided, for example, horizontally and vertically divided into three equal parts, and the three horizontal positions are described as "upper", "middle", and "lower", and the three vertical positions are described as "left", "middle", and "right", that is, divided into a 3×3 grid, and the positions from the upper left to the lower right will be described as "upper left", "upper middle", "upper right", "middle left", "middle", "middle right", "lower left", "lower middle", and "lower right". Then the main category of each area is evaluated; if the main category feature area of ​​any area is no less than 50%, this category is used to describe the area. If the proportion of the main category in any of the three areas is less than 50%, the image is further segmented into finer granularity.

[0050] The VLM text encoding module encodes the text information in the adaptive text generation module and generates a text feature vector T with a dimension of 1×d;

[0051] The convolution module performs convolution processing on the feature image output by the encoder in the final stage to generate the converted image features;

[0052] A global average pooling module: performing average pooling processing on a transformed feature output by the convolution module to generate an image feature vector I of the same dimension 1×d as the transformed feature; and

[0053] The image-text comparison module generates an image-text matrix based on the text feature vector and the image feature vector, and calculates the similarity comparison entropy loss Litc used to generate the similarity score for image-text mutual conversion. This is another loss function of the network of the present invention.

[0054] Embodiment 2:

[0055] like Figure 2As shown, this embodiment is a further design based on the first embodiment, wherein the sub-feature fusion module is divided into two categories: a low-level sub-feature fusion module for fusing underlying image features and a high-level sub-feature fusion module for fusing advanced image features. When image features are extracted sequentially in four stages, the first half of the four stages, stage 1 (stage1) and stage 2 (stage2), use a low-level sub-feature fusion module to extract low-level features of the image, such as edge and contour features, and use convolution-based feature fusion. The second half of the four stages, stage 3 (stage3) and stage 4 (stage4), use a high-level sub-feature fusion module to fuse the extracted image features. The model is located deeper, usually to extract advanced features such as shapes and relationships, which require high-precision aggregation.

[0056] Embodiment three:

[0057] This embodiment is a further design based on the second embodiment, wherein the low-level sub-feature fusion module adopts a convolution-based structure and uses a convolution mechanism to perform image fusion. The specific expression is as follows:

[0058]

[0059] For stage1 and stage2, the image fusion equation Ffusion is mainly a feature fusion equation based on convolution. Conv1×1(·) and Conv5×5 refer to 1×1 and 5×5 convolution calculations. Concate(·, ·, ·, ·,) is the four new feature maps Fc1, Fc2, Fc3, and Fc4 generated by splicing four image features.

[0060] Embodiment 4:

[0061] This embodiment is a further design based on the third embodiment, wherein the high-level sub-feature fusion module adopts an attention-based mechanism, and the specific expression is as follows:

[0062] K = Concate(Flatten(F c2 ),Flatten(F c3 ),Flatten(F c4 ))

[0063] V = Concate(Flatten(F c2 ),Flatten(F c3 ),Flatten(F c4 ))

[0064] Q = Flatten (F c1 )

[0065] Where:

[0066] K represents the key matrix, which usually comes from the feature map of the previous stage or related stage. It stores the feature information compared with the query matrix Q. During feature fusion, the key matrix K is dot-producted with the query matrix Q to generate a similarity score matrix. This matrix represents the similarity between the query feature and the key feature, and is an important basis for subsequent feature fusion.

[0067] V represents the value matrix, which comes from the feature map in the setting stage and stores the feature information of the final output;

[0068] Q represents the query matrix, which is derived from the feature map of the current stage or the set stage, represents the feature information of this stage, and serves as the basis for "querying" or "finding" relevant features;

[0069] In the feature fusion process, the query matrix Q is used to measure the similarity with the key matrix, so as to decide which features should be focused on. Through the query, the model can understand the feature content of the current stage and perform more accurate feature fusion in the subsequent stages;

[0070] Flatten(·) refers to the operation of splicing the obtained 2D feature image into a one-dimensional vector. Flatten(·) refers to the operation of splicing the obtained 2D feature image into a one-dimensional vector;

[0071] Concate(·,·,·,) means concatenating the three feature maps in the brackets.

[0072] Embodiment five:

[0073] This embodiment is a further design based on the first embodiment, wherein the cross entropy loss Lce generated by pixel-by-pixel classification training is expressed as follows:

[0074]

[0075] Where: N represents the number of pixels in the image, and C represents the total number of categories. For each pixel i, represents the true label using one-hot encoding, that is, if pixel i belongs to category c, then otherwise refers to the predicted probability that pixel i belongs to category c, is the logarithm of the predicted probability, which measures the difference between the prediction and the actual label. The closer the predicted probability is to the true label (i.e. The smaller the cross entropy loss is, the smaller the

[0076] Embodiment six:

[0077] This embodiment is a further design based on the first embodiment, in which image-text conversion is performed, that is, image-to-text and text-to-image, and the similarity scores of image-to-text and text-to-image after softmax normalization are calculated, using the following expression:

[0078]

[0079] Where s(·,·) represents the cosine similarity between two vectors, H represents the cross entropy function, and given a text T m The similarity score of image I when p t2i (T) is a given image I m In practice, the familiarity score of text T is calculated by the mean; I refers to image features, T refers to text features; τ is the temperature coefficient; and M represents the total number of texts.

[0080] Embodiment seven:

[0081] This embodiment is a further design based on the first embodiment, wherein the similarity comparison entropy loss Litc is expressed as follows:

[0082]

[0083] In the formula, represents the cross entropy function, represents the expected value, y i2t Represents the true label of the text, i2t refers to the i-th pixel to the m-th text, and t2i refers to the distance from the m-th text to the i-th pixel.

[0084] Application examples:

[0085] like Figure 3 As shown, it is the test result on the test data set according to the present invention. Given that the URUR data set contains significantly more images than the GID data set, and both are composed of ultra-high resolution (UHR) satellite images, the effect of random multi-cropping becomes more significant as the amount of data increases. This may be because multi-scale random cropping effectively expands the amount of data from which the model can learn. Although a certain amount of resolution accuracy is lost during image downsampling, this enables the model to learn more intrinsic features of surface cover objects, which is more critical than learning contextual features. For small data sets with limited data, the learned features are often more single and have poor generalization capabilities. By introducing text-image contrast learning, it is essentially injecting text information from the multimodal model into the semantic segmentation model, thereby enhancing its generalization ability.

[0086] The technical solutions of the present invention are not limited to the above-mentioned embodiments, and any technical solutions obtained by equivalent replacement methods fall within the scope of protection required by the present invention.

Claims

1. A text knowledge guided context feature fusion network for ultra-high resolution satellite images, wherein the ultra-high resolution satellite images are freely multi-scale cropped into multiple scaled images of different scales, and the scaled images are semantically segmented and annotated to form label images, characterized in that: The network includes: An image encoder, comprising a plurality of stage encoders adapted to the scale-adjusted image in number, and the image features of the different scale-adjusted images are extracted by the corresponding plurality of stage encoders in a sequence of gradually increasing depth and gradually decreasing area in a plurality of stages, and each stage encoder obtains a plurality of feature images of the corresponding stage; The feature fusion module includes a plurality of sub-feature fusion modules whose number matches that of the stage encoder and respectively correspond to one stage encoder, each sub-feature fusion module receives a plurality of feature images provided by the encoder of the corresponding stage, and fuses a plurality of features to obtain a feature fusion image of the corresponding stage, and the feature fusion images of the corresponding stages respectively obtained by the plurality of sub-feature fusion modules form a feature fusion image set; The image decoder accepts the feature fusion image set, reconstructs the multi-image features, generates a reconstructed image, and calculates the cross entropy loss Lce generated by the pixel-by-pixel classification training of the input image; The adaptive text generation module accepts the label image and automatically generates the category text of the corresponding image for the full-width category area not less than the set width ratio according to the semantic segmentation of the label image. The image is divided into regions for the category area not less than the set ratio. The category text of the corresponding region is automatically generated for each region with a category area not less than the set domain ratio. If any category area of ​​the three regions is less than the set domain ratio, a more fine-grained regional division is performed. The VLM text encoding module encodes the text information in the adaptive text generation module and generates a text feature vector T with a dimension of 1×d; The convolution module performs convolution processing on the feature image output by the encoder in the final stage to generate the converted image features; A global average pooling module: performing average pooling processing on a transformed feature output by the convolution module to generate an image feature vector I of the same dimension 1×d as the transformed feature; The image-text comparison module generates an image-text matrix based on the text feature vector and the image feature vector, and calculates the similarity comparison entropy loss Litc used to generate the similarity score for image-text conversion.

2. The text knowledge guided context feature fusion network for ultra-high resolution satellite images according to claim 1, characterized in that: The multiple rescaled images are annotated with semantic segmentation, one of which maintains the original ground-truth distance and the others provide contextual information.

3. The text knowledge guided context feature fusion network for ultra-high resolution satellite images according to claim 1, characterized in that: The multiple sub-feature fusion modules are divided into two categories: a low-level sub-feature fusion module for fusing underlying image features and a high-level sub-feature fusion module for fusing advanced image features. When the image features are extracted sequentially in multiple stages, the first half of the multiple stages uses the low-level sub-feature fusion module to fuse the extracted image features, and the second half uses the high-level sub-feature fusion module to fuse the extracted image features.

4. The text knowledge guided context feature fusion network for ultra-high resolution satellite images according to claim 3 is characterized by: The low-level sub-feature fusion module is based on the convolution structure and adopts the following convolution mechanism expression: F fusion =Conv1×1(Conv5×5(Concate(F c1 ,F c2 ,F c3 ,F c4 ))) Where: F fusion represents the image fusion equation; Conv1×1(·) and Conv5×5 represent 1×1 and 5×5 convolution calculations; Concate(·,·,·,·,) means concatenating four image features; Fc1, Fc2, Fc3, and Fc4 represent four new feature maps generated by concatenation, where c represents the category and C is an integer. Here c1 to c4 refer to four categories.

5. The text knowledge guided context feature fusion network for ultra-high resolution satellite images according to claim 3, characterized in that: The high-level sub-feature fusion module is based on the attention mechanism and uses the following expression: K=Concate(Flatten(F c2 ),Flatten(F c3 ),Flatten(F c4 )) V=Concate(Flatten(F c2 ),Flatten(F c3 ),Flatten(F c4 )) Where: K represents the key matrix, Q represents the query matrix, V represents the value matrix, Flatten(·) represents the operation of concatenating the obtained 2D feature images into a one-dimensional vector, and Concate(·,·,·,) represents the operation of concatenating the 3 feature maps in the brackets.

6. The text knowledge guided context feature fusion network for ultra-high resolution satellite images according to claim 1, characterized in that: The cross entropy loss Lce generated by the pixel-by-pixel classification training is expressed as follows: Where: N represents the number of pixels in the image, and C represents the total number of categories. For each pixel i, represents the true label using one-hot encoding, that is, if pixel i belongs to category c, then otherwise refers to the predicted probability that pixel i belongs to category c.

7. The text knowledge guided context feature fusion network for ultra-high resolution satellite images according to claim 1, characterized in that: The image-text conversion similarity score is the similarity score between image and text and text and image after calculating the softmax normalization, using the following expression: Where s(·,·) represents the cosine similarity between two vectors, and p i2t (I) is the given text T m The similarity score of image I when p t2i (T) is a given image I m is the recognition score of text T at that time. In practice, the mean is calculated. I refers to image features, T refers to text features, τ is the temperature coefficient, and M represents the total number of texts.

8. The text knowledge guided context feature fusion network for ultra-high resolution satellite images according to claim 7, characterized in that: The similarity comparison entropy loss Litc is expressed as follows: In the formula, represents the cross entropy function, represents the expected value, y i2t Represents the true label of the text, i2t refers to the i-th pixel to the m-th text, and t2i refers to the distance from the m-th text to the i-th pixel.

Images