Remote sensing image feature extraction and fusion method and device

Abstract

The invention discloses a remote sensing image feature extraction and fusion method and device. The method comprises the following steps: acquiring a plurality of remote sensing images in a target area; inputting each remote sensing image into a pre-constructed multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network; for each extracted feature map, calculating the importance weight of each branch at different pixel positions through a pixel-level self-attention mechanism to obtain a weight matrix; and based on the weight matrix, performing weighted fusion on the feature map of each branch to obtain a fused feature map. Therefore, the method and the device realize intelligent perception of multi-source remote sensing image quality and optimize a feature extraction and fusion algorithm so as to achieve the purposes of shielding low-quality features and merging high-quality features, thereby improving the effect of feature extraction and fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image feature extraction and fusion method and device. Background Art

[0002] At present, deep learning methods have been widely used in various tasks in the field of remote sensing, such as land cover and land use classification, object detection and identification, and natural disaster monitoring and assessment. With its powerful feature extraction capabilities, deep learning can automatically learn rich spatial and spectral information from remote sensing images, greatly improving processing efficiency and accuracy. However, the effectiveness of deep learning models is highly dependent on the feature quality of the input images. The quality of the input images directly affects the accuracy of the model's feature extraction, which in turn affects the performance of downstream tasks.

[0003] In actual remote sensing applications, image quality is often affected by multiple factors such as sensor performance, meteorological conditions (such as clouds and fog, atmospheric interference), and terrain occlusion, and there may be large differences. Especially in areas with complex environments such as mountains and polar regions, due to undulating terrain, drastic changes in lighting conditions, and frequent extreme weather, the quality of remote sensing images is usually poor, accompanied by a lot of noise and distortion. In this case, a single image often cannot provide enough feature information, and noise will significantly affect the feature extraction process of the deep learning model, resulting in large errors in subsequent classification, recognition and other tasks. Due to these factors, the performance of deep learning models is greatly limited, especially when dealing with tasks with high precision requirements, the generalization ability and recognition accuracy of the model are difficult to reach the expected level.

[0004] In order to solve the problem of limited performance of deep learning models caused by the above-mentioned image quality differences, image synthesis is the most widely used method to address the quality differences of multi-source remote sensing images, but it has obvious limitations. Usually, image synthesis uses statistical fusion, weighted averaging and other methods to combine image information obtained at different times or from different sensors in order to enhance image quality. However, when faced with serious image quality differences, this type of method has the following technical problems:

[0005] 1) Interference of low-quality images on synthesis results: When processing images with significant quality differences, image synthesis technology cannot effectively distinguish between high-quality and low-quality images. Low-quality images (such as images obscured by clouds or affected by noise) are often processed together during the synthesis process, resulting in noise and blurred information being mixed in the synthesis results. This method not only fails to improve the accuracy of features, but may also introduce redundant information, affecting the overall performance of subsequent tasks (such as classification, segmentation, etc.).

[0006] 2) Lack of fine perception and weighted processing of image quality: Existing image synthesis methods usually do not effectively perceive and weight differences in image quality. All images are regarded as equal inputs, and the information in high-quality images cannot be fully utilized. When processing multi-source images, this simple fusion strategy will cause the advantages of high-quality images to be masked by low-quality images, thus affecting the final feature extraction effect. Summary of the invention

[0007] To this end, the embodiments of the present invention provide a remote sensing image feature extraction and fusion method and device, in order to achieve intelligent perception of the quality of multi-source remote sensing images, optimize the feature extraction and fusion algorithms, so as to shield low-quality features and merge high-quality features, thereby improving the effect of feature extraction and fusion.

[0008] In order to achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0009] The present invention provides a remote sensing image feature extraction and fusion method, the method comprising:

[0010] Acquire multiple remote sensing images within the target area;

[0011] Inputting each of the remote sensing images into a pre-constructed multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network;

[0012] For each extracted feature map, the importance weight of each branch at different pixel positions is calculated through the pixel-level self-attention mechanism to obtain the weight matrix;

[0013] Based on the weight matrix, the feature map of each branch is weightedly fused to obtain a fused feature map.

[0014] In some embodiments, each of the remote sensing images is input into a pre-built multi-branch feature extraction model, which also includes:

[0015] Determine the target band or target phase;

[0016] Aligning the remote sensing images to ensure that the remote sensing images cover the same target area;

[0017] Each of the remote sensing images is cropped to a target specification to ensure that the resolution of each of the remote sensing images is consistent.

[0018] In some embodiments, the pre-built multi-branch feature extraction network specifically includes:

[0019] A block segmentation layer, the block segmentation layer is used to receive each of the remote sensing images and segment the remote sensing images to obtain a plurality of small-area remote sensing image blocks;

[0020] A first shared extraction layer, the first shared extraction layer is used to extract basic features of each remote sensing image, each remote sensing image shares model parameters in the first shared extraction layer, and the first shared extraction layer includes a linear embedding module and a Swin Transformer Block; the input of the linear embedding module is a segmented block of each remote sensing image, and it is projected to an arbitrary dimension C to output a feature map; the input of the Swin Transformer Block is an output feature map of the linear embedding module, and the output is a processed feature map;

[0021] The second shared extraction layer is used to extract basic features of the feature map corresponding to each of the remote sensing images output by the first shared extraction layer, and each of the remote sensing images shares model parameters in the second shared extraction layer. The second shared extraction layer includes a tile merging module and a Swin Transformer Block; the tile merging module merges four adjacent tiles and outputs a tile with a dimension of 2C; the Swin Transformer Block uses the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map;

[0022] The first independent extraction layer is used to extract the complex features of each feature map output by the second shared extraction layer. The parameters of each remote sensing image in the first independent extraction layer are independent and the model parameters are not shared. The first independent extraction layer includes a tile merging module and a Swin Transformer Block. The tile merging module merges four adjacent tiles to obtain a tile with a dimension of 4C. The Swin Transformer Block uses the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map.

[0023] The second independent extraction layer is used to extract the complex features of the feature maps output by each of the first independent extraction layers, and output the final feature maps corresponding to each of the remote sensing images. The parameters of each of the remote sensing images in the second independent extraction layer are independent and do not share model parameters. The second independent extraction layer includes a tile merging module and a SwinTransformer Block. The tile merging module takes the output feature map of the first independent extraction layer as input, merges four adjacent tiles, and obtains a tile with a dimension of 8C. The Swin Transformer Block takes the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map.

[0024] In some embodiments, the Swin Transformer Block includes multi-head self-attention, residual connection, layer normalization and multi-layer perceptron; within each window, the features are calculated through the self-attention mechanism, followed by residual connection and normalization processing, and finally feature mapping and dimension adjustment are performed through the multi-layer perceptron.

[0025] In some embodiments, the expression of the feature map corresponding to the remote sensing image is:

[0026] F i =Swin2Tiny(I i ),i=1,2,…,n

[0027] Among them, F i is the feature map extracted by the i-th branch, Swin2Tiny represents a multi-branch feature extraction network, I i is the i-th remote sensing image.

[0028] In some embodiments, the expression for calculating the importance weight of each branch at different pixel positions through the pixel-level self-attention mechanism is:

[0029]

[0030] Among them, α i (x,y) is the weight of the i-th branch at pixel position (x,y), Q i (x,y) is the query vector, K i (x,y) is the key vector, indicating the similarity at a specific position.

[0031] In some embodiments, the expression of the fused feature map is:

[0032]

[0033] Among them, F fused (x,y) is the feature map after the pixel position (x,y) is fused, F i (x,y) is the feature map extracted by the i-th branch at pixel position (x,y).

[0034] The present invention also provides a remote sensing image feature extraction and fusion device, the device comprising:

[0035] An image acquisition unit, used to acquire multiple remote sensing images within a target area;

[0036] A feature extraction unit, used for inputting each of the remote sensing images into a pre-constructed multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network;

[0037] A weight calculation unit is used to calculate the importance weight of each branch at different pixel positions for each extracted feature map through a pixel-level self-attention mechanism to obtain a weight matrix;

[0038] The feature fusion unit is used to perform weighted fusion on the feature map of each branch based on the weight matrix to obtain a fused feature map.

[0039] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0041] The remote sensing image feature extraction and fusion method and device provided by the present invention obtain multiple remote sensing images in the target area; input each of the remote sensing images into a pre-constructed multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network; for each extracted feature map, the importance weight of each branch at different pixel positions is calculated through a pixel-level self-attention mechanism to obtain a weight matrix; based on the weight matrix, the feature map of each branch is weightedly fused to obtain a fused feature map. In this way, the method and device realize intelligent perception of the quality of multi-source remote sensing images, optimize the feature extraction and fusion algorithm to shield low-quality features and merge high-quality features, thereby improving the effect of feature extraction and fusion, and further providing more accurate and reliable feature support for subsequent remote sensing tasks such as classification, target detection, and semantic segmentation, and improving the performance and generalization performance of the model in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0043] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0044] Figure 1 One of the flow charts of the remote sensing image feature extraction and fusion method provided by the present invention;

[0045] Figure 2 The second flowchart of the remote sensing image feature extraction and fusion method provided by the present invention;

[0046] Figure 3 A network architecture diagram of a multi-branch feature extraction network provided by the present invention;

[0047] Figure 4 The network structure diagram of the Swin Transformer Block provided by the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of the remote sensing image feature extraction and fusion device provided by the present invention;

[0049] Figure 6 This is a structural block diagram of a computer device provided by the present invention. DETAILED DESCRIPTION

[0050] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In order to solve the problems existing in the prior art, the method provided by the present invention is applicable to various remote sensing tasks that apply deep learning methods, aiming to deal with the quality differences of multi-source images, integrate images of different qualities, extract and fuse high-quality deep features in the images, and provide support for subsequent classification, detection, segmentation and other different deep learning modules. This method intelligently perceives the quality of multi-source images, distinguishes high-quality and low-quality areas, shields noise and incomplete information, and dynamically adjusts the contribution of each image feature to ensure that the extracted features are more accurate and robust, and adapt to complex tasks under different imaging conditions.

[0052] In a specific embodiment, Figure 1 As shown, the remote sensing image feature extraction and fusion method provided by the present invention comprises the following steps:

[0053] S110: Acquire multiple remote sensing images in the target area; that is, input multiple remote sensing images from different sources or of different qualities. Assume that the input image is I 1 ,I 2 ,…,I n , where n is the number of input images, and each image may have different qualities due to different factors such as clouds, fog, and lighting.

[0054] S120: Input each of the remote sensing images into a pre-constructed multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network; for example, for each input image, a multi-branch hierarchical sharing + independent feature extraction network based on SwinTransformer V2 Tiny (Swin2Tiny) is designed to extract the feature map of each image.

[0055] S130: For each extracted feature map, the importance weight of each branch at different pixel positions is calculated through a pixel-level self-attention mechanism to obtain a weight matrix;

[0056] S140: Based on the weight matrix, perform weighted fusion on the feature map of each branch to obtain a fused feature map.

[0057] In this embodiment, the overall architecture provided by the method consists of a multi-branch feature extraction module, a pixel-level self-attention mechanism, and a feature fusion module. Figure 2 As shown in the figure, first, the network inputs images of different sources or different qualities into a multi-branch network, and each branch receives a set of images independently; the overall network design is hierarchical sharing + independent feature extraction. In the initial stage, each branch shares the feature extractors of the first few layers, and extracts basic features through a hierarchical sharing mechanism to reduce the amount of calculation and ensure that each branch retains the consistency of basic features. Subsequently, the network enters the independent feature extraction stage, and each branch remains independent in the subsequent deep feature extraction process, and performs specific extraction for the quality and features of different images to ensure that more key information can be extracted from high-quality images, and the noise features of low-quality images are controlled. After the feature extraction is completed, the network screens and weights the features of each branch through a pixel-level self-attention mechanism. The self-attention mechanism can intelligently perceive the feature quality of a specific area in each branch, dynamically adjust the feature weight of each branch, ensure that high-quality features dominate, and minimize the interference of low-quality features. Finally, the features of each branch are fused after screening, and a more accurate and robust global feature representation is generated by preferentially extracting high-quality information.

[0058] In some embodiments, each of the remote sensing images is input into a pre-built multi-branch feature extraction model, which also includes:

[0059] Determine the target band or target phase;

[0060] Aligning the remote sensing images to ensure that the remote sensing images cover the same target area;

[0061] Each of the remote sensing images is cropped to a target specification to ensure that the resolution of each of the remote sensing images is consistent.

[0062] That is to say, before inputting the image into the extraction model, it is necessary to select a specific band or phase according to the task requirements, then align the images to ensure that they cover the same area, and crop the images to the size required by the network to ensure consistent resolution.

[0063] In step S120, the pre-built multi-branch feature extraction network specifically includes:

[0064] A block segmentation layer, the block segmentation layer is used to receive each of the remote sensing images and segment the remote sensing images to obtain a plurality of small-area remote sensing image blocks;

[0065] The first shared extraction layer is used to extract the basic features of each remote sensing image. Each remote sensing image shares model parameters in the first shared extraction layer. The first shared extraction layer includes a linear embedding module and a Swin Transformer Block (a visual recognition model based on the Transformer architecture). The input of the linear embedding module is the segmented block of each remote sensing image, and it is projected to an arbitrary dimension C to output a feature map. The input of the Swin Transformer Block is the output feature map of the linear embedding module, and the output is the processed feature map;

[0066] The second shared extraction layer is used to extract the basic features of the feature map corresponding to each of the remote sensing images output by the first shared extraction layer. Each of the remote sensing images shares model parameters in the second shared extraction layer. The second shared extraction layer includes a tile merging module and a Swin Transformer Block. The tile merging module merges four adjacent tiles and outputs a tile with a dimension of 2C. The Swin Transformer Block uses the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map;

[0067] The first independent extraction layer is used to extract the complex features of each feature map output by the second shared extraction layer. The parameters of each remote sensing image in the first independent extraction layer are independent and do not share model parameters. The first independent extraction layer includes a tile merging module and a Swin Transformer Block. The tile merging module then merges four adjacent tiles to obtain a tile with a dimension of 4C. The Swin Transformer Block uses the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map;

[0068] The second independent extraction layer is used to extract the complex features of the feature maps output by each first independent extraction layer, and output the final feature maps corresponding to each remote sensing image. The parameters of each remote sensing image in the second independent extraction layer are independent and do not share model parameters. The second independent extraction layer includes a tile merging module and a SwinTransformer Block. The tile merging module takes the output feature map of the first independent extraction layer as input, merges 4 adjacent tiles, and obtains a tile with a dimension of 8C. The Swin Transformer Block takes the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map.

[0069] Each input image I i Corresponding to an independent branch, the expression of the feature map corresponding to the remote sensing image is:

[0070] F i =Swin2Tiny(I i ),i=1,2,…,n

[0071] Among them, F i is the feature map extracted by the i-th branch, Swin2Tiny represents a multi-branch feature extraction network, I i is the i-th remote sensing image.

[0072] Specifically, Figure 3As shown in the figure, Swin2Tiny is calculated with a design of layered sharing + independent feature extraction. The first two layers (Stage1 and Stage2) are responsible for extracting basic features of the image, such as low-level information such as edges and textures. These features are relatively consistent in images of different qualities, so parameters can be shared in all branches. This can not only effectively reduce the computational complexity, but also ensure the unified processing of basic information. However, in the subsequent Stage3 and Stage4, the network begins to extract higher-level complex features, such as target shape and semantic information. At this stage, the quality difference of the images becomes more significant. High-quality images can extract more details and key information, while low-quality images may be full of noise or lack details. Therefore, the feature extraction of these two layers is performed independently in each branch, with the same structure but independent parameters. By independently processing high-level features, it is possible to avoid the noise of low-quality images interfering with the key information of high-quality images, thereby ensuring that subsequent feature screening and fusion are more accurate, and effectively improving the quality and robustness of the final feature representation.

[0073] The method provided by the present invention can independently process remote sensing image features from different sources and of different qualities through a multi-branch structure, avoiding confusion between high-quality and low-quality features. This independent processing method makes feature screening more effective, ensuring that high-quality features can be accurately distinguished and preferentially selected in the subsequent feature fusion process. The method provided by the present invention adopts a hierarchical shared feature extraction architecture, sharing feature extractors in the first few layers to reduce computational complexity, and independently processing high-level feature extraction of each branch in the latter layers. This design ensures that image quality differences are fully considered in deep feature extraction, taking into account both computational efficiency and feature accuracy.

[0074] Furthermore, the Swin Transformer Block includes multi-head self-attention, residual connection, layer normalization and multi-layer perceptron; within each window, the features are calculated through the self-attention mechanism, followed by residual connection and normalization processing, and finally feature mapping and dimension adjustment are performed through the multi-layer perceptron.

[0075] The function of Swin Transformer Block is to extract local and global features of the image through the self-attention mechanism in the local window (Window-based Self-Attention). Its structure is as follows Figure 4As shown in Figure 1. Its structure includes components such as multi-head self-attention, residual connection, LayerNorm (layer normalization) and MLP (multi-layer perceptron). In each window, the features are calculated through the self-attention mechanism, followed by residual connection and normalization, and finally feature mapping and dimension adjustment through MLP. This design can not only capture local details of the image, but also aggregate global information through cross-window connections. q, k, and v represent query, key, and value, respectively, which are connected through W Q , W K , W V A linear projection is performed to map the input features to the attention space. Then, the dot product of the query and the key is used to obtain the attention weight, which is normalized by softmax to weight the information of the value (v). This process is completed through the self-attention mechanism within the window to capture local features. b is a learnable bias term used to enhance the relative position encoding. In the whole process, z is the weighted attention output, which then enters the residual connection and is normalized by LayerNorm to stabilize the training and maintain the consistency of the features. Then, it is further nonlinearly mapped through MLP (multi-layer perceptron) to enhance the expressiveness of the features, and finally residually connected with the input features to obtain the final output X I .

[0076] This method dynamically weights and fuses the features of each branch through a pixel-level self-attention mechanism. This mechanism can automatically adjust the feature weight of each pixel according to the image quality of each branch, ensuring that the features of high-quality images dominate the fusion process and the noise impact of low-quality images is effectively weakened. Compared with the fixed or simple weighting method in the prior art, the fusion mechanism of this method is more flexible and can significantly improve the accuracy and robustness of feature fusion.

[0077] In step S130, for each extracted feature map F i The importance weight of each branch at different pixel positions is calculated through the pixel-level self-attention mechanism. The self-attention mechanism dynamically adjusts the weight of each pixel point according to the feature quality of the image. The expression of the importance weight of each branch at different pixel positions calculated by the pixel-level self-attention mechanism is:

[0078]

[0079] Among them, α i (x,y) is the weight of the i-th branch at pixel position (x,y), Q i (x,y) is the query vector, K i (x,y) is the key vector, indicating the similarity at a specific position.

[0080] After completing the weight calculation, the weight matrix α generated by the pixel-level self-attention mechanism is i (x, y), perform weighted fusion on the feature maps of each branch, and the expression of the fused feature map is:

[0081]

[0082] Among them, F fused (x,y) is the feature map after the pixel position (x,y) is fused, F i (x,y) is the feature map extracted by the i-th branch at pixel position (x,y).

[0083] The fused feature map F fused As the final output of the entire network, it is used for subsequent remote sensing tasks (such as classification, target detection, semantic segmentation, etc.). The output feature map not only contains the key information from multi-source images, but also filters the noise of low-quality images through the self-attention mechanism.

[0084] In the above specific implementation, the remote sensing image feature extraction and fusion method provided by the present invention obtains multiple remote sensing images in the target area; inputs each of the remote sensing images into a pre-constructed multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network; for each extracted feature map, the importance weight of each branch at different pixel positions is calculated through a pixel-level self-attention mechanism to obtain a weight matrix; based on the weight matrix, the feature map of each branch is weightedly fused to obtain a fused feature map. In this way, the method realizes intelligent perception of the quality of multi-source remote sensing images, optimizes feature extraction and fusion algorithms, so as to shield low-quality features and merge high-quality features, thereby improving the effect of feature extraction and fusion, and further providing more accurate and reliable feature support for subsequent remote sensing tasks such as classification, target detection, and semantic segmentation, and improving the performance and generalization performance of the model in complex environments.

[0085] In addition to the above method, the present invention also provides a remote sensing image feature extraction and fusion device, such as Figure 5 As shown, the device comprises:

[0086] An image acquisition unit 510 is used to acquire multiple remote sensing images within a target area;

[0087] A feature extraction unit 520 is used to input each of the remote sensing images into a pre-built multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network;

[0088] A weight calculation unit 530 is used to calculate the importance weight of each branch at different pixel positions for each extracted feature map through a pixel-level self-attention mechanism to obtain a weight matrix;

[0089] The feature fusion unit 540 is used to perform weighted fusion on the feature map of each branch based on the weight matrix to obtain a fused feature map.

[0090] In some embodiments, each of the remote sensing images is input into a pre-built multi-branch feature extraction model, which also includes:

[0091] Determine the target band or target phase;

[0092] Aligning the remote sensing images to ensure that the remote sensing images cover the same target area;

[0093] Each of the remote sensing images is cropped to a target specification to ensure that the resolution of each of the remote sensing images is consistent.

[0094] In some embodiments, the pre-built multi-branch feature extraction network specifically includes:

[0095] The tile segmentation layer is used to receive each of the remote sensing images and segment the remote sensing images to obtain a plurality of small-area remote sensing image tiles.

[0096] The first shared extraction layer is used to extract the basic features of each remote sensing image. Each remote sensing image shares model parameters in the first shared extraction layer. The first shared extraction layer includes a linear embedding module and a Swin Transformer Block. The input of the linear embedding module is the segmented block of each remote sensing image, and it is projected to an arbitrary dimension C to output a feature map. The input of the Swin Transformer Block is the output feature map of the linear embedding module, and the output is the processed feature map;

[0097] The second shared extraction layer is used to extract the basic features of the feature map corresponding to each of the remote sensing images output by the first shared extraction layer. Each of the remote sensing images shares model parameters in the second shared extraction layer. The second shared extraction layer includes a tile merging module and a Swin Transformer Block. The tile merging module merges four adjacent tiles and outputs a tile with a dimension of 2C. The Swin Transformer Block uses the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map;

[0098] The first independent extraction layer is used to extract the complex features of each feature map output by the second shared extraction layer. The parameters of each remote sensing image in the first independent extraction layer are independent and do not share model parameters. The first independent extraction layer includes a tile merging module and a Swin Transformer Block. The tile merging module then merges four adjacent tiles to obtain a tile with a dimension of 4C. The Swin Transformer Block uses the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map;

[0099] The second independent extraction layer is used to extract the complex features of the feature maps output by each of the first independent extraction layers, and output the final feature maps corresponding to each of the remote sensing images. The parameters of each of the remote sensing images in the second independent extraction layer are independent and do not share model parameters. The second independent extraction layer includes a tile merging module and a SwinTransformer Block. The tile merging module takes the output feature map of the first independent extraction layer as input, merges four adjacent tiles, and obtains a tile with a dimension of 8C. The Swin Transformer Block takes the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map;

[0100] In some embodiments, the Swin Transformer Block includes multi-head self-attention, residual connection, layer normalization and multi-layer perceptron; within each window, the features are calculated through the self-attention mechanism, followed by residual connection and normalization processing, and finally feature mapping and dimension adjustment are performed through the multi-layer perceptron.

[0101] In some embodiments, the expression of the feature map corresponding to the remote sensing image is:

[0102] F i =Swin2Tiny(I i ),i=1,2,…,n

[0103] Among them, F i is the feature map extracted by the i-th branch, Swin2Tiny represents a multi-branch feature extraction network, I i is the i-th remote sensing image.

[0104] In some embodiments, the expression for calculating the importance weight of each branch at different pixel positions through the pixel-level self-attention mechanism is:

[0105]

[0106] Among them, α i(x,y) is the weight of the i-th branch at pixel position (x,y), Q i (x,y) is the query vector, K i (x,y) is the key vector, indicating the similarity at a specific position.

[0107] In some embodiments, the expression of the fused feature map is:

[0108]

[0109] Among them, F fused (x,y) is the feature map after the pixel position (x,y) is fused, F i (x,y) is the feature map extracted by the i-th branch at pixel position (x,y).

[0110] In the above specific implementation, the remote sensing image feature extraction and fusion device provided by the present invention obtains multiple remote sensing images in the target area; inputs each of the remote sensing images into a pre-constructed multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network; for each extracted feature map, the importance weight of each branch at different pixel positions is calculated through a pixel-level self-attention mechanism to obtain a weight matrix; based on the weight matrix, the feature map of each branch is weightedly fused to obtain a fused feature map. In this way, the device realizes intelligent perception of the quality of multi-source remote sensing images, optimizes feature extraction and fusion algorithms, so as to shield low-quality features and merge high-quality features, thereby improving the effect of feature extraction and fusion, and further providing more accurate and reliable feature support for subsequent remote sensing tasks such as classification, target detection, and semantic segmentation, and improving the performance and generalization performance of the model in complex environments.

[0111] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a model prediction. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The model prediction of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0112] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0113] Corresponding to the above embodiment, the embodiment of the present invention further provides a computer storage medium, which contains one or more program instructions, wherein the one or more program instructions are used to execute the above method.

[0114] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the above method.

[0115] In the embodiment of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0116] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0117] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0118] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0119] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0120] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0121] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0122] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A remote sensing image feature extraction and fusion method, characterized in that: The method comprises: Acquire multiple remote sensing images within the target area; Inputting each of the remote sensing images into a pre-constructed multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network; For each extracted feature map, the importance weight of each branch at different pixel positions is calculated through the pixel-level self-attention mechanism to obtain the weight matrix; Based on the weight matrix, the feature map of each branch is weightedly fused to obtain a fused feature map.

2. The remote sensing image feature extraction and fusion method according to claim 1, characterized in that: Inputting each of the remote sensing images into a pre-built multi-branch feature extraction model, previously including: Determine the target band or target phase; Aligning the remote sensing images to ensure that the remote sensing images cover the same target area; Each of the remote sensing images is cropped to a target specification to ensure that the resolution of each of the remote sensing images is consistent.

3. The remote sensing image feature extraction and fusion method according to claim 1, characterized in that: Pre-built multi-branch feature extraction network, including: A block segmentation layer, the block segmentation layer is used to receive each of the remote sensing images and segment the remote sensing images to obtain a plurality of small-area remote sensing image blocks; A first shared extraction layer, the first shared extraction layer is used to extract basic features of each remote sensing image, each remote sensing image shares model parameters in the first shared extraction layer, and the first shared extraction layer includes a linear embedding module and a Swin Transformer Block; wherein the input of the linear embedding module is a segmented block of each remote sensing image, and the block is projected to an arbitrary dimension C to output a feature map; the input of the Swin Transformer Block is an output feature map of the linear embedding module, and the output is a processed feature map; A second shared extraction layer, wherein the second shared extraction layer is used to extract basic features of feature maps corresponding to each of the remote sensing images output by the first shared extraction layer, and each of the remote sensing images shares model parameters in the second shared extraction layer, and the second shared extraction layer includes a tile merging module and a Swin Transformer Block; wherein the tile merging module merges four adjacent tiles and outputs a tile with a dimension of 2C; the Swin Transformer Block uses the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map; The first independent extraction layer is used to extract the complex features of each feature map output by the second shared extraction layer. The parameters of each remote sensing image in the first independent extraction layer are independent and the model parameters are not shared. The first independent extraction layer includes a tile merging module and a Swin Transformer Block. The tile merging module merges four adjacent tiles to obtain a tile with a dimension of 4C. The Swin Transformer Block uses the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map. The second independent extraction layer is used to extract the complex features of the feature maps output by each first independent extraction layer, and output the final feature maps corresponding to each remote sensing image. The parameters of each remote sensing image in the second independent extraction layer are independent and do not share model parameters. The second independent extraction layer includes a tile merging module and a SwinTransformer Block. The tile merging module takes the output feature map of the first independent extraction layer as input, merges 4 adjacent tiles, and obtains a tile with a dimension of 8C. The Swin Transformer Block takes the output of the tile merging module as input, performs feature conversion on the tile to extract parameters, and outputs a feature map.

4. The remote sensing image feature extraction and fusion method according to claim 3, characterized in that: SwinTransformer Block includes multi-head self-attention, residual connection, layer normalization and multi-layer perceptron; within each window, the features are calculated through the self-attention mechanism, followed by residual connection and normalization, and finally feature mapping and dimension adjustment through the multi-layer perceptron.

5. The remote sensing image feature extraction and fusion method according to claim 3, characterized in that: The expression of the feature map corresponding to the remote sensing image is: F i =Swin2Tiny(I i ),i=1,2,…,n Among them, F i is the feature map extracted by the i-th branch, Swin2Tiny represents a multi-branch feature extraction network, I i is the i-th remote sensing image.

6. The remote sensing image feature extraction and fusion method according to claim 1, characterized in that: The expression of the importance weight of each branch at different pixel positions is calculated through the pixel-level self-attention mechanism: Among them, α i (x,y) is the weight of the i-th branch at pixel position (x,y), Q i (x,y) is the query vector, K i (x,y) is the key vector, indicating the similarity at a specific position.

7. The remote sensing image feature extraction and fusion method according to claim 6, characterized in that: The expression of the fused feature map is: Among them, F fused (x,y) is the feature map after the pixel position (x,y) is fused, F i (x,y) is the feature map extracted by the i-th branch at pixel position (x,y).

8. A remote sensing image feature extraction and fusion device, characterized in that: The device comprises: An image acquisition unit, used to acquire multiple remote sensing images within a target area; A feature extraction unit, used for inputting each of the remote sensing images into a pre-constructed multi-branch feature extraction network to obtain a feature map corresponding to each remote sensing image output by the multi-branch feature extraction network; A weight calculation unit is used to calculate the importance weight of each branch at different pixel positions for each extracted feature map through a pixel-level self-attention mechanism to obtain a weight matrix; The feature fusion unit is used to perform weighted fusion on the feature map of each branch based on the weight matrix to obtain a fused feature map.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images