Remote sensing image processing method and device, electronic equipment, storage medium and product

Abstract

The invention discloses a remote sensing image processing method and device, electronic equipment, a storage medium and a product. The method comprises the steps that a shallow feature extraction module in an image processing model is adopted to generate first feature information of a first remote sensing image, the shallow feature extraction module is configured with a plurality of depth separable convolves which are used in parallel, and the plurality of depth separable convolves are respectively provided with convolution kernels of different sizes; a deep feature extraction module in an image processing model is adopted to extract second feature information from the first feature information, wherein the second feature information is abstract and semantic expression of contour features and texture features represented by the first feature information; and according to the second feature information and the first remote sensing image, performing image reconstruction by using an image reconstruction module in an image processing model to obtain a second remote sensing image. According to the scheme, the problems of flat region interference and insufficient image feature utilization in the remote sensing image are solved, and the precision and efficiency of the super-resolution of the remote sensing image are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a remote sensing image processing method, device, electronic equipment, storage medium and product. Background Art

[0002] Remote sensing image super-resolution technology reconstructs low-resolution images to restore high-resolution details, which is of great significance for improving image quality in fields such as Earth observation, environmental monitoring, and disaster warning. However, traditional deep learning-based remote sensing image super-resolution methods face significant technical bottlenecks when processing complex scenes. This is mainly reflected in the typical scene characteristics of remote sensing images, which often show a mixture of large flat areas and complex high-frequency textures (such as the coexistence of homogeneous areas such as farmland and water bodies with heterogeneous textures such as buildings and forests). Traditional deep learning-based remote sensing image super-resolution methods generally lack the ability to capture multi-scale features in the shallow feature extraction stage. In particular, they tend to over-allocate attention resources in large flat areas and ignore the recovery of high-frequency details. Summary of the Invention

[0003] The present invention provides a remote sensing image processing method, device, electronic device, storage medium and product to solve the problems of flat area interference and insufficient utilization of image features in remote sensing images, and improve the accuracy and efficiency of remote sensing image super-resolution.

[0004] According to one aspect of the present invention, a remote sensing image processing method is provided, wherein the method comprises:

[0005] Generating first feature information of a first remote sensing image using a shallow feature extraction module in an image processing model, wherein the shallow feature extraction module is configured with a plurality of depthwise separable convolutions used in parallel, wherein the plurality of depthwise separable convolutions respectively have convolution kernels of different sizes, wherein the first feature information includes contour features and texture features extracted for objects of different sizes in the first remote sensing image using the convolution kernels of different sizes;

[0006] Extracting second feature information from the first feature information using a deep feature extraction module in the image processing model, where the second feature information is an abstract and semantic expression of the contour features and texture features represented by the first feature information;

[0007] According to the second feature information and the first remote sensing image, an image reconstruction module in the image processing model is used to perform image reconstruction to obtain a second remote sensing image.

[0008] According to another aspect of the present invention, a remote sensing image processing device is provided, wherein the device comprises:

[0009] a shallow feature extraction module, configured to generate first feature information of the first remote sensing image using the shallow feature extraction module in the image processing model, wherein the shallow feature extraction module is configured with multiple depthwise separable convolutions used in parallel, each of the multiple depthwise separable convolutions having convolution kernels of different sizes, and the first feature information includes contour features and texture features extracted for objects of different sizes in the first remote sensing image using the convolution kernels of different sizes;

[0010] a deep feature extraction module, configured to extract second feature information from the first feature information using the deep feature extraction module in the image processing model, wherein the second feature information is an abstract and semantic expression of the contour features and texture features represented by the first feature information;

[0011] An image reconstruction module is used to reconstruct an image using the image reconstruction module in the image processing model according to the second feature information and the first remote sensing image to obtain a second remote sensing image.

[0012] According to another aspect of the present invention, an electronic device is provided, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the remote sensing image processing method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the remote sensing image processing method according to any embodiment of the present invention when executed.

[0017] According to another aspect of the present invention, a computer program product is provided, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the remote sensing image processing method according to any embodiment of the present invention is implemented.

[0018] The technical solution of the embodiment of the present invention is that in the shallow feature extraction link, parallel multi-size depth-separable convolution kernels can synchronously capture the first feature information containing multi-resolution contour edges and texture details for objects of different scales such as vehicles, buildings, and farmland in the first remote sensing image, thereby realizing the extraction of multi-scale features of high-frequency and low-frequency areas of images with different texture complexities. Compared with a single convolution kernel, it solves the problem of insufficient detail capture in high-frequency areas, significantly improves the comprehensiveness of feature extraction, and avoids the problem of small target features being ignored or large target details being blurred; at the same time, in the deep feature extraction link, through hierarchical abstraction, the first feature information of shallow pixel-level features is converted into second feature information of semantic concepts, realizing the leap from low-level vision to high-level semantics, retaining the clarity of semantic boundaries during reconstruction, and enabling the model to understand the geographical meaning of the image content; further, by combining the first feature information with the first remote sensing image through the image reconstruction module, a second remote sensing image with enhanced spatial resolution and clearer semantic structure can be generated.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a remote sensing image processing method provided according to an embodiment of the present invention;

[0022] Figure 2 is an architectural diagram of a remote sensing image processing process applicable to an embodiment of the present invention;

[0023] Figure 3 is an architectural diagram of a shallow feature extraction module in a remote sensing image processing process applicable to an embodiment of the present invention;

[0024] Figure 4 is a flowchart of another remote sensing image processing method provided by an embodiment of the present invention;

[0025] Figure 5 is a flowchart of another remote sensing image processing method provided according to an embodiment of the present invention;

[0026] Figure 61 is an architectural diagram of a window attention integrated block in a deep feature extraction module in a remote sensing image processing process applicable to an embodiment of the present invention;

[0027] Figure 7 is an architectural diagram of a hybrid attention group in a deep feature extraction module in a remote sensing image processing process applicable to an embodiment of the present invention;

[0028] Figure 8 is a schematic diagram of an adaptive feature fusion layer in a remote sensing image processing process applicable to an embodiment of the present invention;

[0029] Figure 9 1 is a schematic structural diagram of a remote sensing image processing device provided according to an embodiment of the present invention;

[0030] Figure 10 The figure is a schematic structural diagram of an electronic device for implementing the remote sensing image processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 A flowchart of a remote sensing image processing method provided in an embodiment of the present invention is provided. This embodiment is applicable to situations where the inherent multi-scale features of remote sensing images are utilized to improve the super-resolution reconstruction of remote sensing images. The remote sensing image processing method can be performed by a remote sensing image processing device, which can be implemented in the form of hardware and / or software. The remote sensing image processing device can be configured in any electronic device with network communication capabilities.

[0034] like Figure 1 As shown, the remote sensing image processing method of this embodiment includes the following steps:

[0035] S110. Generate first feature information of the first remote sensing image using a shallow feature extraction module in the image processing model. The shallow feature extraction module is configured with multiple depthwise separable convolutions used in parallel. The multiple depthwise separable convolutions respectively have convolution kernels of different sizes. The first feature information includes contour features and texture features extracted for objects of different sizes in the first remote sensing image using convolution kernels of different sizes.

[0036] The first remote sensing image may be image data acquired through remote sensing technology from a remote sensing platform at high altitude or in outer space, using sensors deployed on the remote sensing platform to detect and perceive objects on the Earth's surface. Remote sensing platforms include, but are not limited to, at least one of the following: satellites, aircraft, and drones. Sensors deployed on the remote sensing platform include, but are not limited to, at least one of the following: cameras, scanners, and radars.

[0037] See also Figure 2 The shallow feature extraction module is a module in the image processing model used to extract basic-level features from the input first remote sensing image. The shallow feature extraction module is configured at the front end of the network structure near the input layer of the image processing model. The shallow feature extraction module is configured to capture low-level and localized features in the first remote sensing image. Low-level and localized features in the first remote sensing image include, but are not limited to, at least one of the following: edge, color, texture, and simple shape.

[0038] See also Figure 3 In the shallow feature extraction module, multiple depth-wise separable convolutions can be configured for parallel use. Multiple depth-wise separable convolutions have convolution kernels of different sizes, which means that different-sized convolution kernels can simultaneously capture detailed structures of different sizes. In this way, by processing input features in parallel through convolution kernels of different sizes, local spatial information of different scales in the first remote sensing image can be captured simultaneously, thereby enhancing the adaptability of the image processing model to multi-scale features.

[0039] Depthwise separable convolution is a lightweight design in convolutional neural networks. By decomposing a standard convolution into a depthwise convolution operation and a pointwise convolution operation, it significantly reduces the amount of computation and parameters required for the first remote sensing image while maintaining or approaching the performance of a standard convolution. Optionally, each of the multiple depthwise separable convolutions can be a 3×3 depthwise separable convolution, a 5×5 depthwise separable convolution, or a 7×7 depthwise separable convolution.

[0040] See also Figure 3, multiple depthwise separable convolution branches run in parallel, and depthwise separable convolutions configured in different branches use convolution kernels of different sizes to process the same input first remote sensing image. Among them, the smaller the size of the convolution kernel, the smaller the receptive field of the convolution kernel, the more emphasis is placed on local details, and the more suitable it is for extracting small-sized object targets (such as vehicles and telephone poles) or finer textures (such as leaf veins and roof tiles) in remote sensing images. Correspondingly, the larger the size of the convolution kernel, the larger the receptive field of the convolution kernel, the more suitable it is for capturing global structures and extracting large-sized object targets (such as lakes and airport runways) in remote sensing images.

[0041] By using multiple depth-wise separable convolutions in parallel to implement convolution operations with multiple different convolution kernel sizes, and fusing the output feature maps of different depth-wise separable convolutions, we can capture multi-scale and multi-receptive field features and improve the image model's ability to express complex inputs.

[0042] Contour features can refer to the geometric shape and spatial distribution of object edges in remote sensing images, such as straight lines, curves, and corners. A superficial manifestation of contour features can be directly calculated from the gradient changes of image pixels, such as using the Sobel operator or Canny edge detection. These features rely solely on the grayscale / color differences between local adjacent pixels and do not involve semantic understanding. For example, contour features can be the smooth edges of lakes in remote sensing images or the circular outlines of cells in medical images.

[0043] Texture features can refer to recurring local patterns and their regular arrangement in remote sensing images. Texture features include, but are not limited to, at least one of the following: roughness, directionality, and periodicity (e.g., the warp and weft textures of fabric or the veins of leaves). A superficial manifestation of texture features can be extracted by statistically analyzing the gray-level co-occurrence matrix or gradient directional histogram of pixels in a local area. This reflects the spatial distribution patterns at the pixel level and does not involve specific semantic understanding. For example, texture features can be the row and column texture of farmland or the granular structure of forest canopies shown in remote sensing images.

[0044] S120. Use a deep feature extraction module in the image processing model to extract second feature information from the first feature information. The second feature information is an abstract and semantic expression of the contour feature and texture feature represented by the first feature information.

[0045] The first feature information is generated by the shallow feature extraction module in the image processing model and includes contour and texture features extracted using convolution kernels of different sizes for objects of different sizes in the first remote sensing image. For example, a remote sensing image may contain contours and textures as small as a vehicle or as large as a mountain or lake.

[0046] The deep feature extraction module extracts features from the first feature information output by the shallow feature extraction module through multiple layers of nonlinear transformations. This allows for a high-level, abstract representation of the first feature information output by the shallow feature extraction module, reflecting the semantic information contained in the first feature information. The second feature information can be an abstracted and semanticized representation of the contour and texture features represented by the first feature information. Abstraction removes specific details, while semanticization transforms features into meaningful concepts. For example, it converts contour and texture information in remote sensing images into semantic information.

[0047] Using this approach, the first feature information consists primarily of low-level contour and texture features, lacking clear semantic information. The second feature information, obtained through the deep feature extraction module, transforms these shallow features into semantically meaningful expressions, enabling the image processing model to better understand the content of remote sensing images and improve image interpretation capabilities. The deep feature extraction module abstracts the first feature information, removing noise and irrelevant details, allowing the image processing model to learn more essential features.

[0048] S130 . Reconstruct an image using an image reconstruction module in an image processing model according to the second feature information and the first remote sensing image to obtain a second remote sensing image.

[0049] The first remote sensing image contains rich raw pixel information, such as the color, texture, and shape of various features. The image reconstruction module is used to integrate the second feature information into the first remote sensing image through feature fusion, based on the input second feature information and the first remote sensing image. This allows the reconstruction of a new remote sensing image with specific features or optimized effects, referred to as the second remote sensing image. By combining the details of the first remote sensing image with the high-level semantic information contained in the second feature information, the second remote sensing image can achieve improved clarity, semantic accuracy, and information completeness compared to the first remote sensing image.

[0050] Optionally, based on the second feature information and the first remote sensing image, image reconstruction is performed using an image reconstruction module in the image processing model, including: feature fusing the second feature information with the first remote sensing image to obtain a feature fusion result, and feature decoding the feature fusion result to gradually restore it to the spatial resolution of the remote sensing image, while retaining and enhancing the semantic information contained in the fused features; converting the feature representation obtained by feature decoding into image pixel values, thereby reconstructing the second remote sensing image.

[0051] Optionally, feature fusion of the second feature information with the first remote sensing image includes: performing a feature alignment operation on the second feature information and the image features of the first remote sensing image, and fusing the second feature information with the image features of the first remote sensing image after the feature alignment. The feature alignment operation includes the following operations: upsampling or downsampling the image features of the first remote sensing image using methods such as bilinear interpolation and nearest neighbor interpolation, so that the image features of the first remote sensing image and the second feature information maintain the same size. Feature fusion includes, but is not limited to, at least one of the following: splicing and fusing the image features of the second feature information and the first remote sensing image in the channel dimension; assigning different weights to the second feature information and the image features of the first remote sensing image and then performing weighted fusion.

[0052] Optionally, the feature representation obtained by feature decoding is converted into image pixel values ​​to reconstruct a second remote sensing image, including: using one or more convolutional layers to process the feature representation obtained by feature decoding after decoding, converting the number of channels of the feature representation obtained by feature decoding into the number of channels of the remote sensing image, the convolution kernel weights of the convolution layer in the image reconstruction module are trained and learned to be able to map the feature representation obtained by feature decoding to the pixel space of the image; and applying an activation function at the output of the convolution layer to limit the output value to a preset range. For image pixel values, the output value can be mapped to the interval [0,1] using a Sigmoid function, or mapped to the interval [-1,1] using a Tanh function, and then scaled according to actual conditions to reconstruct the second remote sensing image.

[0053] Optionally, after the image reconstruction module in the image processing model is used to perform image reconstruction based on the second feature information and the first remote sensing image to obtain the second remote sensing image, the method further includes at least one of the following:

[0054] The second remote sensing image is smoothed using at least one of Gaussian filtering and median filtering to remove noise and salt and pepper spots in the second remote sensing image, thereby making the second remote sensing image smoother.

[0055] The contrast of the second remote sensing image is enhanced by at least one equalization method selected from the group consisting of histogram equalization and adaptive histogram equalization, so that details of the second remote sensing image are more clearly visible.

[0056] The color of the second remote sensing image is corrected to make the color of the second remote sensing image more realistic and natural; for example, the color correction can be achieved by adjusting parameters such as hue, saturation, and brightness of the second remote sensing image.

[0057] Using this approach, the high-level semantic information contained in the second feature information is integrated into the second remote sensing image, ensuring that the remote sensing image not only retains the original pixel details but also has clear semantic meaning. By utilizing the second feature information to adjust and optimize the first remote sensing image, the image reconstruction module can remove noise, blur, and other issues in the image, improving image quality and clarity. The first remote sensing image may contain some areas with missing or incomplete information, such as areas obscured by clouds. The image reconstruction module can repair and supplement these missing areas based on the second feature information and surrounding contextual information, making the smoke image more complete and thus better ensuring that the reconstructed second remote sensing image has richer semantic information and higher image quality.

[0058] As an optional but non-limiting implementation, the image processing model is generated using, but not limited to, the following methods:

[0059] A training set of an image processing model is obtained, and the image processing model is trained and updated according to the training set of the image processing model, where the training set includes a fourth remote sensing image generated by performing bicubic downsampling on the third remote sensing image and adding noise, where the third remote sensing image is a remote sensing image with a resolution greater than a preset value.

[0060] A batch of high-resolution remote sensing images is collected or generated as the third remote sensing images, ensuring that their resolution exceeds a preset value (e.g., 1 meter / pixel). Each third remote sensing image is sequentially subjected to bicubic downsampling and noise addition to generate a fourth remote sensing image. The fourth remote sensing images can then be input into the image processing model as a training set. The model is then trained and updated through the shallow feature extraction module, deep feature extraction module, and image reconstruction module. The third remote sensing images are high-resolution remote sensing images with a resolution exceeding a preset threshold.

[0061] Bicubic downsampling is an image resampling technique that generates a lower-resolution image by calculating the weighted average of the 16 pixels surrounding the target pixel. The weighting coefficient is determined by a cubic interpolation function. Compared with nearest neighbor and bilinear interpolation, bicubic downsampling better preserves image details and reduces aliasing artifacts. Bicubic downsampling can simulate the degradation of remote sensing images in real scenes caused by sensor resolution limitations or transmission compression in the training set, generating low-quality images as input samples.

[0062] Adding noise can introduce random interference signals (such as Gaussian noise and salt-and-pepper noise) into the third-party remote sensing images, simulating noise contamination caused by electromagnetic interference, sensor errors, and other factors during remote sensing image acquisition and transmission. By adding noise, the diversity and authenticity of training data are enhanced, allowing the model to learn more robust features and improve its ability to handle real-world noisy images.

[0063] For example, training and test sets are selected from the AID, DOTAv1.0, DIOR, and NWPU-RESISC45 datasets, where the training set contains 3,000 high-resolution remote sensing images and the test set contains 920 images. Bicubic downsampling is performed on the images in the training and test sets to generate low-resolution image samples (scaling factors ×2 / ×3 / ×4), and Gaussian noise (σ=5) and JPEG compression (quality factor 30) are added to simulate real degradation, thereby obtaining the training set of the image processing model.

[0064] As another optional but non-limiting implementation, the image processing model is generated using, but not limited to, the following methods:

[0065] The image processing model is trained and updated according to a pre-set model training method; the model training method includes the following implementation process: in the first training stage, the image processing model is optimized by minimizing the pixel loss function L1 loss, and the learning rate is decayed by 50% every 10,000 iterations based on the configured initial learning rate; and, in the second training stage, the image processing model is fine-tuned by using the MSE function, and the learning rate is decayed by 50% every 200,000 iterations based on the configured initial learning rate.

[0066] The first training phase is the initial phase of image processing model training, focusing on optimizing pixel-level detail reconstruction of remote sensing images. This can be achieved by configuring the loss function, initial learning rate, and learning rate decay as follows: The loss function of the first training phase uses L1 loss, also known as mean absolute error (MAE), which calculates the average of the absolute values ​​of the differences between the predicted remote sensing image and the real remote sensing image. The initial learning rate of the first training phase is configured as The learning rate decay configuration in the first training phase is that the learning rate decays by 50% after every 10,000 iterations of training. The image processing model iterates 100,000 times in the first training phase to avoid oscillation caused by too high a learning rate in the early stage and slow convergence caused by too low a learning rate in the later stage. The specific loss function is: Where N is the total number of pixels, yi is the predicted value of the i-th pixel, and GTi is the true value of the i-th pixel.

[0067] The second training stage is the refinement stage of image processing model training. It optimizes the semantic structure and overall quality of the image based on pixel-level reconstruction, and improves the visual authenticity of the reconstructed image through a smoother loss function. This can be achieved by configuring the loss function, initial learning rate, and learning rate decay as follows: The loss function of the second training stage uses the MSE function (mean square error) to calculate the average value of the square of the difference between the corresponding pixel values ​​of the predicted remote sensing image and the real remote sensing image; the initial learning rate of the second training stage is configured to 10 −4The learning rate decay configuration in the first training stage is that the learning rate decays by 50% after every 10,000 iterations of training. The image processing model iterates 200,000 times in the first training stage to meet the needs of fine-tuning model parameters in the later stage and avoid falling into the local optimum too early. The specific loss function is: Where N is the total number of pixels, yi is the predicted value of the i-th pixel, and GTi is the true value of the i-th pixel.

[0068] During the first training phase, the gradient of the L1 loss is constant (invariant to the error), resulting in drastic parameter updates at the beginning of training. This allows for rapid reduction of pixel-level differences between the predicted and true images, making it suitable for processing high-percentage background areas in remote sensing images. Furthermore, the L1 loss is insensitive to outliers in individual pixels, helping the model prioritize learning the overall structure early in training and avoid local noise interference. During the first training phase, the gradient of the MSE loss is proportional to the error, resulting in stronger updates for areas with large pixel differences (such as object edges and texture details). This guides the model to optimize high-frequency details in the reconstructed image (such as building edges and vegetation textures), improving visual realism. Furthermore, through smooth gradient updates, the model more easily learns the global structural semantics of the image.

[0069] Furthermore, the initial learning rate in the first training phase is high, resulting in large parameter updates. High-frequency decay rapidly reduces the learning rate, preventing the model from oscillating around the local optimum due to excessive learning rates. This also forces the model to focus on detailed fitting early in training. The initial learning rate in the second training phase is low, resulting in small parameter updates. Low-frequency decay maintains the learning rate stable over a longer period, allowing the model to make subtle adjustments to parameters and raising the upper limit of reconstruction accuracy. Through staged loss function switching and dynamic learning rate decay, progressive optimization is achieved, from pixel-level coarse adjustment to semantic-level refinement.

[0070] The technical solution of the embodiment of the present invention is that in the shallow feature extraction link, parallel multi-size depth-separable convolution kernels can synchronously capture the first feature information containing multi-resolution contour edges and texture details for objects of different scales such as vehicles, buildings, and farmland in the first remote sensing image, thereby realizing the extraction of multi-scale features of high-frequency and low-frequency areas of images with different texture complexities. Compared with a single convolution kernel, it solves the problem of insufficient detail capture in high-frequency areas, significantly improves the comprehensiveness of feature extraction, and avoids the problem of small target features being ignored or large target details being blurred; at the same time, in the deep feature extraction link, through hierarchical abstraction, the first feature information of shallow pixel-level features is converted into second feature information of semantic concepts, realizing the leap from low-level vision to high-level semantics, retaining the clarity of semantic boundaries during reconstruction, and enabling the model to understand the geographical meaning of the image content; further, by combining the first feature information with the first remote sensing image through the image reconstruction module, a second remote sensing image with enhanced spatial resolution and clearer semantic structure can be generated.

[0071] Figure 4 This is a flowchart of another remote sensing image processing method provided by an embodiment of the present invention. This embodiment is further optimized on the basis of the above embodiment, including but not limited to further optimizing the process of generating the first feature information of the first remote sensing image using the shallow feature extraction module in the above image processing model. This embodiment can be combined with various optional schemes in one or more of the above embodiments.

[0072] like Figure 4 As shown, the remote sensing image processing method of the embodiment of the present invention may include the following steps:

[0073] S410. Use the depth-separable convolutions of different branches in the shallow feature extraction module in the image processing model to extract features of the first remote sensing image respectively, and obtain image features output by the separable convolutions of different branch depths.

[0074] Among them, the shallow feature extraction module is configured with multiple depth-wise separable convolutions used in parallel, and the multiple depth-wise separable convolutions respectively have convolution kernels of different sizes. The first feature information includes contour features and texture features extracted by convolution kernels of different sizes for objects of different sizes in the first remote sensing image.

[0075] Remote sensing images often exhibit typical scene characteristics characterized by a mixture of large flat areas and complex high-frequency textures (e.g., homogeneous areas such as farmland and water bodies coexist with heterogeneous textures such as buildings and forests). Using a single-scale convolution kernel in the shallow feature extraction stage results in insufficient multi-scale feature capture. In particular, for large flat areas, single-scale convolutions can easily overallocate attention resources, resulting in computational redundancy and a failure to effectively suppress low-frequency noise. For complex high-frequency textures (e.g., vegetation structures and building outlines at different resolutions), the receptive field of a single-scale convolution fails to encompass cross-scale feature dependencies, leading to inadequate detail recovery and problems such as blurred edges and texture fragmentation in the reconstructed image. To address this issue, depthwise separable convolutions with different branches are employed to extract features from the first remote sensing image. In the shallow feature extraction stage, convolution kernels of different sizes are used to capture the image features in the first remote sensing image, yielding image features output by separable convolutions with different branch depths.

[0076] See also Figure 3Taking the depthwise separable convolutions of different branches (3×3, 5×5, and 7×7) as an example, the first remote sensing image is input into the depthwise separable convolutions of different branches of the shallow feature extraction module. Each branch performs convolution using a convolution kernel of its corresponding size: the first branch uses a depthwise separable convolution with a 3×3 kernel, the second branch uses a 5×5 kernel, and the third branch uses a 7×7 kernel. Depthwise separable convolutions of different sizes have different receptive fields and can extract feature information of different scales from the first remote sensing image. After the first remote sensing image passes through the depthwise separable convolutions of each branch, the image features of each branch's depthwise separable convolution are output.

[0077] S420: splicing the image features outputted by the separable convolutions of different branch depths to obtain spliced ​​image features.

[0078] S430: Activate the spliced ​​image features through a GELU activation function to generate multi-scale features of the first remote sensing image, where the multi-scale features of the first remote sensing image include feature information extracted from the first remote sensing image at different scales.

[0079] See also Figure 3 , the image features output by separable convolutions of different branch depths can be spliced ​​in the channel dimension. The purpose of this is to integrate feature information of different scales together to form a feature representation containing richer information.

[0080] See also Figure 3 , the spliced ​​image features are activated through the GELU activation function to generate the multi-scale features of the first remote sensing image. The spliced ​​image features are input into the GELU activation function for activation. The activation function performs a nonlinear transformation on each element in the spliced ​​image features, so that the image processing model can learn more complex feature representations. After processing by the GELU activation function, the multi-scale features of the first remote sensing image are generated.

[0081] The multi-scale features of the first remote sensing image can reflect information of different sizes and levels of detail in the first remote sensing image. Different objects in remote sensing images can have different sizes. For example, satellite remote sensing images contain both large-scale objects like cities and small-scale objects like small buildings and vehicles. Multi-scale feature extraction can simultaneously capture the characteristic properties of these objects of different sizes. Multi-scale features can also integrate information at different scales, providing a richer and more comprehensive feature representation.

[0082] By using depthwise separable convolution kernels of different sizes in different branches, feature information at different scales can be extracted from the first remote sensing image. Small-scale convolution kernels can capture detailed image features, such as edges and textures; large-scale convolution kernels can capture the overall structure and semantic information of the image, such as the shape and layout of objects. Concatenating these different-scale feature information and activating them with the GELU activation function enhances the expressive power of the features. Using depthwise separable convolution in the shallow feature extraction module can reduce the model's computational cost and improve its operational efficiency without significantly sacrificing performance.

[0083] By using the GELU activation function, image processing models can learn more complex feature representations, thereby improving model performance. Extracting multi-scale features enables the model to better recognize objects of varying sizes. Remote sensing images may contain objects of varying sizes, such as small buildings and large lakes. By extracting multi-scale features, the model can adapt to the characteristic variations of objects of varying sizes, improving its generalization and reducing the risk of overfitting.

[0084] As an optional but non-limiting implementation, see Figure 3 , the image features output by the separable convolutions of different branch depths are spliced ​​to obtain the spliced ​​image features, including but not limited to the following steps A1-A2:

[0085] Step A1: According to the local gradient amplitude of the first remote sensing image, the weight ratios of convolution kernels of different sizes are configured. The weight ratios of convolution kernels of different sizes are used to indicate the importance of image features output by separable convolutions of different branch depths and to indicate the contribution of image features in the transmission process. The local gradient amplitude of the image is used to reflect the severity of pixel changes in local areas of the image.

[0086] Step A2: According to the weight ratio of convolution kernels of different sizes, the image features output by separable convolutions of different branch depths are spliced ​​to obtain spliced ​​image features.

[0087] The gradient amplitude of high-frequency areas in remote sensing images (such as edges, textures, and small targets) is large, requiring a small-size convolution kernel to capture local details; however, the gradient amplitude of low-frequency areas in remote sensing images (such as smooth backgrounds and large object surfaces) is small, which is suitable for large-size convolution kernels to extract global semantics, and redundancy can be reduced by reducing the computational intensity of large kernels.

[0088] Using fixed-weight multi-scale convolutions applies kernels of different sizes equally to all regions. This leads to two major problems: first, detail is lost in high-frequency areas. Large convolution kernels (e.g., 7×7) can blur details in high-frequency areas such as edges and textures (the receptive field covers multiple edge pixels, resulting in gradient averaging). Second, computational redundancy occurs in low-frequency areas. In low-frequency areas with smooth backgrounds (such as the sky and water), large kernels (e.g., 7×7) fail to provide effective information, wasting computing power. To address this issue, the weight ratios of convolution kernels of different sizes can be dynamically adjusted based on the local gradient magnitude of the first remote sensing image. The local gradient magnitude can be calculated by using a gradient operator (e.g., the Sobel operator) to calculate the rate of change of pixel intensity in a local region of the first remote sensing image. This reflects the richness of detail or edge saliency in the local region of the first remote sensing image.

[0089] First, local areas of the image with high local gradient amplitude in the remote sensing image (such as object edges and texture-dense areas) require small-size convolution kernels to capture details. Small-size convolution kernels dominate feature extraction due to their high weight ratio, capture pixel-level edges and textures, and avoid large kernels blurring details; local areas of the image with low local gradient amplitude in the remote sensing image (such as smooth background areas) are more suitable for large-size convolution kernels to extract the overall structure. Large-size convolution kernels extract global structure due to their high weight ratio, reducing the noise sensitivity problem of small kernels.

[0090] A method that dynamically adjusts the weights of multi-scale convolution kernels based on the local gradient amplitude of the image. By sensing the structural characteristics of the image content (high-frequency details or low-frequency smooth areas), it adaptively allocates computing resources of convolution kernels of different sizes, which can not only enhance the ability to restore details but also reduce redundant calculations.

[0091] As an optional but non-limiting implementation, see Figure 3 The convolution kernels of different sizes include at least the first convolution kernel and the second convolution kernel. The first convolution kernel is the smallest convolution kernel among the convolution kernels of different sizes, and the second convolution kernel is the largest convolution kernel among the convolution kernels of different sizes. In the area where the gradient amplitude becomes higher, the weight of the first convolution kernel is increased and the weight of the second convolution kernel is suppressed. In the area where the gradient amplitude becomes lower, the weight of the first convolution kernel is suppressed and the weight of the second convolution kernel is increased.

[0092] See also Figure 3In the first remote sensing image, local regions with high local gradient magnitudes (such as edges, textures, and small objects) require a first convolution kernel (such as a 3×3 kernel) to capture local details. In the first remote sensing image, local regions with low local gradient magnitudes (such as smooth backgrounds and large object surfaces) are suitable for a second convolution kernel (such as a 7×7 kernel) to extract global semantics, and this can reduce redundancy by lowering the computational intensity of the first convolution kernel. Therefore, for local regions with high local gradient magnitudes, the weight of the first convolution kernel is increased, while the weight of the second convolution kernel is suppressed. For regions with low local gradient magnitudes, the weight of the second convolution kernel is appropriately increased, and computational complexity can be reduced through lightweight design (such as sparse computing and channel pruning).

[0093] S440: Determine first feature information of the first remote sensing image according to the multi-scale features of the first remote sensing image.

[0094] Optionally, determining the first feature information of the first remote sensing image based on the multi-scale features of the first remote sensing image includes: merging the multi-scale features of the first remote sensing image with the image features of the first remote sensing image to obtain the first feature information of the first remote sensing image.

[0095] Exemplarily, in the shallow feature extraction module, three sets of parallel depth-wise separable convolution structures (with convolution kernel sizes of 3×3, 5×5, and 7×7, respectively) are used to achieve efficient capture of multi-scale features. The specific process is as follows: First, each branch processes the image features of the input first remote sensing image through depth-wise separable convolution of the corresponding size, and divides the image features output by the separable convolutions of different branch depths into three equal parts along the channel dimension to form multiple channel block features. Subsequently, the channel block features of different branches are cross-scale interacted, and the channel block features from each branch are sequentially spliced ​​to form a mixed feature map containing multi-branch and multi-scale information, thereby obtaining the spliced ​​image features. Next, the GELU activation function is introduced into the spliced ​​image features to enhance the nonlinear expression, generate the multi-scale features of the first remote sensing image that integrates the multi-scale context, and use 1×1 convolution to perform cross-channel aggregation on the multi-scale features of the first remote sensing image. For example, the interaction process of the 3×3 branch is as follows: , then the process of generating the shallow feature map is: The shallow features of objects of different sizes are captured through parallel multi-scale depth-wise separable convolution, providing hierarchical input for the subsequent attention mechanism.

[0096] S450. Use a deep feature extraction module in the image processing model to extract second feature information from the first feature information. The second feature information is an abstract and semantic expression of the contour feature and texture feature represented by the first feature information.

[0097] S460: Based on the second feature information and the first remote sensing image, an image reconstruction module in the image processing model is used to perform image reconstruction to obtain a second remote sensing image.

[0098] The technical solution of the embodiment of the present invention is that in the shallow feature extraction link, parallel multi-size depth-separable convolution kernels can synchronously capture the first feature information containing multi-resolution contour edges and texture details for objects of different scales such as vehicles, buildings, and farmland in the first remote sensing image, thereby realizing the extraction of multi-scale features of high-frequency and low-frequency areas of images with different texture complexities. Compared with a single convolution kernel, it solves the problem of insufficient detail capture in high-frequency areas, significantly improves the comprehensiveness of feature extraction, and avoids the problem of small target features being ignored or large target details being blurred; at the same time, in the deep feature extraction link, through hierarchical abstraction, the first feature information of shallow pixel-level features is converted into second feature information of semantic concepts, realizing the leap from low-level vision to high-level semantics, retaining the clarity of semantic boundaries during reconstruction, and enabling the model to understand the geographical meaning of the image content; further, by combining the first feature information with the first remote sensing image through the image reconstruction module, a second remote sensing image with enhanced spatial resolution and clearer semantic structure can be generated.

[0099] Figure 5 This is a flowchart of another remote sensing image processing method provided by an embodiment of the present invention. This embodiment is further optimized on the basis of the above embodiment, including but not limited to further optimizing the process of extracting the second feature information from the first feature information using the deep feature extraction module in the above image processing model. This embodiment can be combined with various optional schemes in one or more of the above embodiments.

[0100] like Figure 5 As shown, the remote sensing image processing method of the embodiment of the present invention may include the following steps:

[0101] S510. Generate first feature information of the first remote sensing image using a shallow feature extraction module in the image processing model. The shallow feature extraction module is configured with multiple depthwise separable convolutions used in parallel. The multiple depthwise separable convolutions respectively have convolution kernels of different sizes. The first feature information includes contour features and texture features extracted by convolution kernels of different sizes for objects of different sizes in the first remote sensing image.

[0102] S520. Divide the feature map corresponding to the first feature information into multiple first windows through the window attention integrated block in the deep feature extraction module of the image processing model. The multiple first windows are multiple non-overlapping windows of preset sizes.

[0103] See also Figure 6The window attention integrated block in the deep feature extraction module is a complete neural network layer or substructure formed by combining window attention with auxiliary operations (such as at least one auxiliary operation among convolution, layer normalization, and residual connection) on the basis of the window attention mechanism. It is used to divide the feature map corresponding to the first feature information into multiple non-overlapping windows and calculate self-attention in each window to reduce the computational complexity.

[0104] S530. Determine the window aggregation feature of each first window, where the window aggregation feature of the first window is obtained by aggregating the image feature vector included in the first window and the image feature vector included in at least one second window among the multiple first windows, and the similarity between the image feature vector included in the at least one second window and the image feature vector included in the first window is greater than the similarity between the image feature vector included in the remaining first windows except the at least one second window among the multiple first windows and the image feature vector included in the first window.

[0105] See also Figure 6 After obtaining multiple non-overlapping first windows, each first window can be regarded as an independent vector. For each first window, the similarity between each first window and the other remaining first windows is calculated. For example, at least one of cosine similarity and Euclidean distance can be used to calculate the window vector similarity between each first window and the other remaining first windows. A preset number of first windows with the highest similarity between each first window and the other first windows are selected from all the first windows as at least one second window. This helps to extract window information of at least one second window that is most representative and relevant to each first window, thereby reducing redundant information.

[0106] Optionally, calculate the cosine similarity matrix between the first windows , Represents the similarity matrix formed by dividing the feature map corresponding to the first feature information into m×n first windows and the similarity between the first windows. A single element Calculated by vector inner product normalization:

[0107]

[0108] Among them, wi and wj are the input features, that is, the feature map after the feature map is divided into several windows, RSTL() refers to the processing by the corresponding module, LN refers to layer normalization, Flatten() refers to flattening it into a one-dimensional vector, and C is the number of channels.

[0109] See also Figure 6, each first window is concatenated with at least one second window corresponding to each selected first window according to the channel dimension; assuming that the dimension of each window is C×H×W (C is the number of channels, H is the height, and W is the width), the dimension of the concatenated tensor will be 5C×H×W. This is done to integrate the information of these five windows in the channel direction so that subsequent operations can process the features of these first windows simultaneously.

[0110] As an optional but non-limiting implementation, see Figure 6 , determining the window aggregation feature of each first window, including but not limited to the following steps:

[0111] Determine at least one second window associated with each first window from multiple first windows; for each first window, splice the image features included in the at least one second window associated with each first window according to the channel dimension and then perform reference convolution to compress the number of channels to obtain the window aggregation features of each first window, and the reference convolution is a convolution operation with a convolution kernel size of 1×1.

[0112] See also Figure 6 The reference convolution is a convolution operation with a convolution kernel size of 1×1. 1×1 convolution is a special form of convolution in deep learning. The image features included in at least one second window associated with each first window are spliced ​​according to the channel dimension and then input into the reference convolution. Its purpose is to compress the number of channels. Through the convolution operation, the number of channels can be compressed to a specified number. While compressing the number of channels, the reference convolution can also realize cross-channel information interaction, allowing the network to automatically learn the combination of different input channel information, increase the nonlinear expression ability of the image processing model, and reduce the number of parameters and computation.

[0113] S540: Determine second feature information included in the first feature information according to the window aggregation feature of each first window in the plurality of first windows.

[0114] The second feature information is an abstract and semantic expression of the contour features and texture features represented by the first feature information.

[0115] Optionally, determining the second feature information included in the first feature information based on the window aggregation feature of each first window in the multiple first windows includes: directly determining the window aggregation feature of each first window in the multiple first windows as the second feature information included in the first feature information.

[0116] As an optional but non-limiting implementation, see Figure 6 , determining the second feature information included in the first feature information according to the window aggregation feature of each first window in the plurality of first windows, including but not limited to the following steps B1-B3:

[0117] Step B1: For each of the multiple first windows, determine the similarity score of each first window. The similarity score of the first window is determined based on whether the first window includes pixels related to the target object and the proportion of the pixels related to the target object in the window area of ​​the first window. The target object is an object that needs to be paid attention to in the first remote sensing image.

[0118] Step B2: determining at least two third windows from the plurality of first windows based on the similarity score of each first window, wherein the similarity scores of the at least two third windows are greater than the similarity scores of the remaining first windows in the plurality of first windows except for the at least two third windows;

[0119] Step B3: Determine the second feature information included in the first feature information based on the window aggregation features of at least two third windows.

[0120] The similarity score of the first window is a quantitative indicator that measures the degree of relevance between the first window and the target object, and its value range is usually [0, 1]. A higher similarity score for the first window indicates a greater probability that the first window contains the target object, or a higher proportion of pixels related to the target object in the first window. Correspondingly, a lower similarity score for the first window indicates a lower probability that the first window contains the target object, or a lower proportion of pixels related to the target object in the first window.

[0121] The third window is selected from all first windows, and the similarity scores of at least two third windows are greater than the similarity scores of the remaining first windows in the plurality of first windows excluding the at least two third windows. The target object can be a feature or target of particular interest in the first remote sensing image, such as a building, vehicle, or ship. Its pixels have unique spectral characteristics (such as the high reflectivity of a building) or spatial morphology (such as the geometric outline of a vehicle).

[0122] Optionally, based on the similarity score of each first window, at least two third windows are determined from multiple first windows, including: sorting all the first windows in descending order according to the similarity scores of the first windows, and directly selecting the top N first windows as third windows, ensuring that the third windows cover the main part of the target object, while excluding windows with a high background ratio.

[0123] Based on the window aggregation features of at least two third windows, determining the second feature information included in the first feature information, including: performing cross-window aggregation on the window aggregation features of at least two third windows, such as by but not limited to performing cross-window aggregation on the window aggregation features of each third window by weighted summation through an attention mechanism to obtain the second feature information included in the first feature information.

[0124] Exemplarily, for each first window including the image feature vector , select the M first windows with the highest similarity as at least two second windows (Top-S strategy), for example, select the 5 first windows with the highest similarity , the image feature vector included in the first window and the image feature vectors included in at least two second windows corresponding to the first window are spliced ​​along the channel dimension and input into the 1×1 convolution to compress the number of channels, obtain the window aggregate feature of the first window, and calculate the similarity score of the first window. For example, the window aggregate feature of the i-th first window adopts the following The aggregation process of the first window is used to obtain:

[0125]

[0126]

[0127] in, ,..., The index of the top five first windows ranked by similarity is retained as the third window based on the first window similarity score. The window aggregation features of each third window are concatenated along the channel dimension to obtain the second feature information included in the first feature information. By capturing texture similarity across windows, the feature representation of repetitive structures (such as farmland and building complexes) is enhanced. Through similarity score screening, only target-related windows (such as building windows) are retained, and the features of background windows (such as grassland and river) are excluded from the aggregation process to prevent irrelevant information from interfering with the extraction of target semantics.

[0128] As an optional but non-limiting implementation scheme, the deep feature extraction module of the image processing model includes a channel attention block, and the window attention integration block and the channel attention block are used to enable the deep feature extraction module in the image processing model to simultaneously focus on spatial local similarity and global channel correlation to prevent the limitations of a single attention mechanism; determining the second feature information included in the first feature information based on the window aggregation feature of each first window of the multiple first windows, including but not limited to the following steps:

[0129] The channel attention block in the deep feature extraction module in the image processing model performs channel attention calculation on the feature map corresponding to the first feature information to determine the inter-channel correlation corresponding to the first feature information; the adaptive feature fusion layer integrates the inter-channel correlation corresponding to the first feature information with the window aggregation features of at least two third windows to obtain the second feature information included in the first feature information.

[0130] See also Figure 7The channel attention block can be a component in the deep feature extraction module used to model the dependencies between channels in the feature map. It is used to calculate the importance weight of each channel, strengthen key semantic channels (such as the near-infrared channel in remote sensing images is sensitive to vegetation), and suppress redundant channels. The window attention integration block focuses on components with local spatial similarity. By dividing non-overlapping windows and aggregating similar window features, it captures the spatial continuity and contextual association of the target object (such as multiple adjacent windows of the same feature in remote sensing images). The adaptive feature fusion layer in the deep feature extraction module of the image processing model dynamically integrates features from different sources (such as spatial window features and channel semantic features). It achieves feature complementarity through adaptive weight adjustment, avoids feature redundancy caused by simple splicing, and automatically balances the contribution of spatial and channel information according to task requirements.

[0131] The Window Attention Integrated Block (WAIB) and Channel Attention Block (CAB) included in the Hybrid Attention Group (MAG) can be combined in the following ways to leverage their respective strengths and improve model performance:

[0132] Serial connection: Input features are first fed into WAIB, where attention calculations are performed within a local window to obtain features processed by window attention, capturing local spatial information. The output is then used as input for CAB, which then performs channel attention calculations on the input features, learns inter-channel dependencies, generates and weights channel attention weights. This allows the model to further consider global channel correlations while focusing on local spatial similarities. Specifically, the input features are first divided into local windows and self-attention is calculated using a window self-attention mechanism (similar to WAIB). Then, the channel attention mechanism (CAB) introduces global information and calculates channel attention weights. Finally, the weighted sum of the two results is added to the input features through a residual connection, achieving better reconstruction.

[0133] Parallel connection: Input features are fed simultaneously into WAIB and CAB, allowing them to independently perform attention calculations in the spatial locality and channel dimensions, respectively, resulting in two distinct attention features. Fusion operations, such as adding or concatenating the two features, can then be used to combine the spatial locality information with the channel global information. Addition simply combines the information from the two attentions, while concatenation increases the feature dimension, providing a richer feature representation for subsequent networks. The fused features are then fed into subsequent network layers for processing.

[0134] See also Figure 7 and Figure 8 , you can deploy the Window Attention Integrated Block (WAIB) and Channel Attention Block (CAB) in parallel. CAB uses a global channel attention mechanism to calculate the inter-channel dependency through the Query-Key matrix:

[0135]

[0136] in, Generated by linear transformation of input features, Represents the vector dimensions used by the query matrix Q and the key matrix K. The outputs of WAIB and CAB are integrated through the adaptive feature fusion layer (AFF): first, the window attention and channel attention are calculated on the input, and the results are integrated using the AFF module, which is expressed as:

[0137] \

[0138] This design enables the model to focus on both spatial local similarity and global channel correlation, avoiding the limitations of a single attention mechanism.

[0139] Attention uses a fixed window for global similarity calculation, which cannot adapt to the distribution characteristics of multi-scale objects in remote sensing images, resulting in insufficient long-range semantic continuity modeling capabilities. In addition, full-window similarity calculation significantly increases memory usage, resulting in a 20%-30% increase in invalid calculations. Therefore, a high-similarity window selection mechanism is used when extracting deep features. By adaptively screening high-value local areas, cross-window interaction is achieved, adapting to the distribution of multi-scale objects, enhancing long-range semantic continuity, and fusing window attention and channel attention to enrich deep feature content.

[0140] S550: Based on the second feature information and the first remote sensing image, an image reconstruction module in the image processing model is used to perform image reconstruction to obtain a second remote sensing image.

[0141] This application scheme has broad application prospects in the field of remote sensing image processing. Its dynamic multi-scale feature fusion and multi-scale window attention mechanism can significantly improve the parsing capabilities of high-resolution satellite images, drone aerial photography and other data, and is particularly suitable for scenarios such as agricultural precision management (such as field ridge boundary identification and crop disease monitoring), water conservancy facility inspection (irrigation channel damage detection), urban planning (land feature classification and change detection) and environmental protection (dynamic tracking of water pollution). Through sparse window screening and low-rank compression technology, the present invention can efficiently process ultra-large images with tens of thousands of pixels, adapt to edge computing devices (such as real-time processing on drones), reduce dependence on cloud computing power, and assist in field surveys, emergency rescue and other tasks with high real-time requirements. In addition, its cross-scene generalization capabilities and plug-and-play characteristics enable it to be seamlessly integrated into existing remote sensing processing platforms (such as ENVI and PIE-Engine), reducing the company's secondary development costs. With the explosive growth of global remote sensing data, this technology is expected to promote industrial upgrades such as intelligent agriculture, digital cities, and refined environmental monitoring, and provide high-precision, low-energy technical support for ecological assessments under the goals of land and resources management and smart city construction.

[0142] The technical solution of the embodiment of the present invention is that in the shallow feature extraction link, parallel multi-size depth-separable convolution kernels can synchronously capture the first feature information containing multi-resolution contour edges and texture details for objects of different scales such as vehicles, buildings, and farmland in the first remote sensing image, thereby realizing the extraction of multi-scale features of high-frequency and low-frequency areas of images with different texture complexities. Compared with a single convolution kernel, it solves the problem of insufficient detail capture in high-frequency areas, significantly improves the comprehensiveness of feature extraction, and avoids the problem of small target features being ignored or large target details being blurred; at the same time, in the deep feature extraction link, through hierarchical abstraction, the first feature information of shallow pixel-level features is converted into second feature information of semantic concepts, realizing the leap from low-level vision to high-level semantics, retaining the clarity of semantic boundaries during reconstruction, and enabling the model to understand the geographical meaning of the image content; further, by combining the first feature information with the first remote sensing image through the image reconstruction module, a second remote sensing image with enhanced spatial resolution and clearer semantic structure can be generated.

[0143] Figure 9 This is a schematic structural diagram of a remote sensing image processing device provided in an embodiment of the present invention. This embodiment is applicable to situations where the inherent multi-scale characteristics of remote sensing images are utilized to improve the super-resolution reconstruction of remote sensing images. The remote sensing image processing device can be implemented in the form of hardware and / or software and can be configured in any electronic device with network communication capabilities.

[0144] like Figure 9 As shown, the remote sensing image processing device of this embodiment includes the following:

[0145] A shallow feature extraction module 910 is configured to generate first feature information of a first remote sensing image using the shallow feature extraction module in the image processing model, wherein the shallow feature extraction module is configured with multiple depthwise separable convolutions used in parallel, each of the multiple depthwise separable convolutions having convolution kernels of different sizes, and the first feature information includes contour features and texture features extracted for objects of different sizes in the first remote sensing image using convolution kernels of different sizes;

[0146] A deep feature extraction module 920 is configured to extract second feature information from the first feature information using the deep feature extraction module in the image processing model, where the second feature information is an abstract and semantic expression of the contour features and texture features represented by the first feature information;

[0147] The image reconstruction module 930 is configured to reconstruct the second remote sensing image using the image reconstruction module in the image processing model according to the second feature information and the first remote sensing image to obtain a second remote sensing image.

[0148] Based on the above embodiment, optionally, generating first feature information of the first remote sensing image using a shallow feature extraction module in an image processing model includes:

[0149] Using depth-separable convolutions of different branches in a shallow feature extraction module in an image processing model to extract features from the first remote sensing image, respectively, to obtain image features output by the separable convolutions of different branch depths;

[0150] Splicing the image features respectively output by the separable convolutions of different branch depths to obtain spliced ​​image features;

[0151] activating the spliced ​​image features through a GELU activation function to generate multi-scale features of the first remote sensing image, where the multi-scale features of the first remote sensing image include feature information extracted from the first remote sensing image at different scales;

[0152] First feature information of the first remote sensing image is determined according to the multi-scale features of the first remote sensing image.

[0153] On the basis of the above embodiment, optionally, the image features outputted by the separable convolutions of different branch depths are spliced ​​to obtain spliced ​​image features, including

[0154] configuring weight ratios of the convolution kernels of different sizes based on the local image gradient amplitude of the first remote sensing image, wherein the weight ratios of the convolution kernels of different sizes are used to indicate the importance of the image features output by the separable convolutions of different branch depths and to indicate the contribution of the image features in the transmission process, and the local image gradient amplitude is used to reflect the severity of pixel changes in a local area of ​​the image;

[0155] According to the weight ratios of the convolution kernels of different sizes, the image features respectively output by the separable convolutions of different branch depths are spliced ​​to obtain spliced ​​image features.

[0156] On the basis of the above embodiment, optionally, the convolution kernels of different sizes include at least a first convolution kernel and a second convolution kernel, the first convolution kernel is the smallest convolution kernel among the convolution kernels of different sizes, and the second convolution kernel is the largest convolution kernel among the convolution kernels of different sizes; in the area where the gradient amplitude becomes higher, the weight of the first convolution kernel is increased, and the weight of the second convolution kernel is suppressed; in the area where the gradient amplitude becomes lower, the weight of the first convolution kernel is suppressed, and the weight of the second convolution kernel is increased.

[0157] Based on the above embodiment, optionally, extracting second feature information from the first feature information using a deep feature extraction module in the image processing model includes:

[0158] Dividing the feature map corresponding to the first feature information into a plurality of first windows by a window attention integrated block in a deep feature extraction module of the image processing model, wherein the plurality of first windows are a plurality of non-overlapping windows of a preset size;

[0159] determining a window aggregate feature of each first window, where the window aggregate feature of the first window is obtained by aggregating an image feature vector included in the first window and an image feature vector included in at least one second window among the multiple first windows, wherein a similarity between the image feature vector included in the at least one second window and the image feature vector included in the first window is greater than a similarity between the image feature vector included in remaining first windows other than the at least one second window among the multiple first windows and the image feature vector included in the first window;

[0160] Second feature information included in the first feature information is determined according to a window aggregation feature of each first window in the plurality of first windows.

[0161] Based on the above embodiment, optionally, determining the window aggregation feature of each first window includes:

[0162] determining at least one second window associated with each first window from a plurality of first windows;

[0163] For each first window, the image features included in at least one second window associated with each first window are spliced ​​according to the channel dimension and then the reference convolution is performed to compress the number of channels to obtain the window aggregation features of each first window. The reference convolution is a convolution operation with a convolution kernel size of 1×1.

[0164] Based on the above embodiment, optionally, determining the second feature information included in the first feature information according to the window aggregation feature of each first window in the multiple first windows includes:

[0165] Determining, for each of the plurality of first windows, a similarity score for each of the first windows, the similarity score of the first window being determined based on whether the first window includes pixels related to a target object and a percentage of a window area in the first window occupied by the pixels related to the target object, the target object being an object of interest in the first remote sensing image;

[0166] determining at least two third windows from the plurality of first windows according to the similarity score of each of the first windows, wherein the similarity scores of the at least two third windows are greater than the similarity scores of the remaining first windows in the plurality of first windows except for the at least two third windows;

[0167] The second feature information included in the first feature information is determined based on the window aggregation features of the at least two third windows.

[0168] Based on the above embodiment, optionally, the deep feature extraction module of the image processing model includes a channel attention block, and the window attention integration block and the channel attention block are used to enable the deep feature extraction module in the image processing model to simultaneously focus on spatial local similarity and global channel correlation to avoid the limitations of a single attention mechanism;

[0169] Determining, according to the window aggregation feature of each first window among the plurality of first windows, second feature information included in the first feature information, includes:

[0170] performing channel attention calculation on the feature map corresponding to the first feature information by a channel attention block in a deep feature extraction module of the image processing model to determine the inter-channel correlation corresponding to the first feature information;

[0171] The inter-channel correlation corresponding to the first feature information and the window aggregation features of the at least two third windows are integrated through an adaptive feature fusion layer to obtain the second feature information included in the first feature information.

[0172] Based on the above embodiment, optionally, the image processing model is generated in at least one of the following ways:

[0173] Obtaining a training set of the image processing model, and training and updating the image processing model according to the training set of the image processing model, wherein the training set includes a fourth remote sensing image generated by performing bicubic downsampling on the third remote sensing image and adding noise, wherein the third remote sensing image is a remote sensing image having a resolution greater than a preset value;

[0174] The image processing model is trained and updated according to a preset model training method; the model training method includes the following implementation process: in a first training stage, the image processing model is optimized by minimizing the pixel loss function L1 loss, and based on the configuration of the initial learning rate, the learning rate decays by 50% every 10,000 iterations; and, in a second training stage, the image processing model is fine-tuned by using the MSE function, and based on the configuration of the initial learning rate, the learning rate decays by 50% every 200,000 iterations.

[0175] The technical solution of the embodiment of the present invention is that in the shallow feature extraction link, parallel multi-size depth-separable convolution kernels can synchronously capture the first feature information containing multi-resolution contour edges and texture details for objects of different scales such as vehicles, buildings, and farmland in the first remote sensing image, thereby realizing the extraction of multi-scale features of high-frequency and low-frequency areas of images with different texture complexities. Compared with a single convolution kernel, it solves the problem of insufficient detail capture in high-frequency areas, significantly improves the comprehensiveness of feature extraction, and avoids the problem of small target features being ignored or large target details being blurred; at the same time, in the deep feature extraction link, through hierarchical abstraction, the first feature information of shallow pixel-level features is converted into second feature information of semantic concepts, realizing the leap from low-level vision to high-level semantics, retaining the clarity of semantic boundaries during reconstruction, and enabling the model to understand the geographical meaning of the image content; further, by combining the first feature information with the first remote sensing image through the image reconstruction module, a second remote sensing image with enhanced spatial resolution and clearer semantic structure can be generated.

[0176] The remote sensing image processing device provided in the embodiment of the present invention can execute the remote sensing image processing method provided in any embodiment of the present invention mentioned above, and has the corresponding functions and beneficial effects of executing the remote sensing image processing method. For detailed process, please refer to the relevant operations of the remote sensing image processing method in the aforementioned embodiment.

[0177] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present invention.

[0178] Figure 10 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0179] like Figure 10As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0180] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0181] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as remote sensing image processing methods.

[0182] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0183] In some embodiments, the remote sensing image processing method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the remote sensing image processing method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the remote sensing image processing method via any other suitable means (e.g., via firmware).

[0184] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0185] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0186] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0187] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0188] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0189] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0190] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0191] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A remote sensing image processing method, characterized in that: The method comprises: Generating first feature information of a first remote sensing image using a shallow feature extraction module in an image processing model, wherein the shallow feature extraction module is configured with a plurality of depthwise separable convolutions used in parallel, wherein the plurality of depthwise separable convolutions respectively have convolution kernels of different sizes, wherein the first feature information includes contour features and texture features extracted for objects of different sizes in the first remote sensing image using the convolution kernels of different sizes; Extracting second feature information from the first feature information using a deep feature extraction module in the image processing model, where the second feature information is an abstract and semantic expression of the contour features and texture features represented by the first feature information; According to the second feature information and the first remote sensing image, an image reconstruction module in the image processing model is used to perform image reconstruction to obtain a second remote sensing image.

2. The method according to claim 1, characterized in that Generating first feature information of the first remote sensing image using a shallow feature extraction module in an image processing model, including: Using depth-separable convolutions of different branches in a shallow feature extraction module in an image processing model to extract features from the first remote sensing image, respectively, to obtain image features output by the separable convolutions of different branch depths; Splicing the image features respectively output by the separable convolutions of different branch depths to obtain spliced ​​image features; activating the spliced ​​image features through a GELU activation function to generate multi-scale features of the first remote sensing image, where the multi-scale features of the first remote sensing image include feature information extracted from the first remote sensing image at different scales; First feature information of the first remote sensing image is determined according to the multi-scale features of the first remote sensing image.

3. The method according to claim 2, characterized in that The image features outputted by the separable convolutions of different branch depths are spliced ​​to obtain spliced ​​image features, including configuring weight ratios of the convolution kernels of different sizes based on the local image gradient amplitude of the first remote sensing image, wherein the weight ratios of the convolution kernels of different sizes are used to indicate the importance of the image features output by the separable convolutions of different branch depths and to indicate the contribution of the image features in the transmission process, and the local image gradient amplitude is used to reflect the severity of pixel changes in a local area of ​​the image; According to the weight ratios of the convolution kernels of different sizes, the image features respectively output by the separable convolutions of different branch depths are spliced ​​to obtain spliced ​​image features.

4. The method according to claim 3, characterized in that The convolution kernels of different sizes include at least a first convolution kernel and a second convolution kernel, the first convolution kernel is the smallest convolution kernel among the convolution kernels of different sizes, and the second convolution kernel is the largest convolution kernel among the convolution kernels of different sizes; in the area where the gradient amplitude becomes higher, the weight of the first convolution kernel is increased and the weight of the second convolution kernel is suppressed; in the area where the gradient amplitude becomes lower, the weight of the first convolution kernel is suppressed and the weight of the second convolution kernel is increased.

5. The method according to claim 1, wherein Extracting second feature information from the first feature information using a deep feature extraction module in the image processing model includes: Dividing the feature map corresponding to the first feature information into a plurality of first windows by a window attention integrated block in a deep feature extraction module of the image processing model, wherein the plurality of first windows are a plurality of non-overlapping windows of a preset size; determining a window aggregate feature of each first window, where the window aggregate feature of the first window is obtained by aggregating an image feature vector included in the first window and an image feature vector included in at least one second window among the multiple first windows, wherein a similarity between the image feature vector included in the at least one second window and the image feature vector included in the first window is greater than a similarity between the image feature vector included in remaining first windows other than the at least one second window among the multiple first windows and the image feature vector included in the first window; Second feature information included in the first feature information is determined according to a window aggregation feature of each first window in the plurality of first windows.

6. The method according to claim 5, characterized in that Determine the window aggregation characteristics of each first window, including: determining at least one second window associated with each first window from a plurality of first windows; For each first window, the image features included in at least one second window associated with each first window are spliced ​​according to the channel dimension and then the reference convolution is performed to compress the number of channels to obtain the window aggregation features of each first window. The reference convolution is a convolution operation with a convolution kernel size of 1×1.

7. The method according to claim 5, characterized in that Determining, according to the window aggregation feature of each first window among the plurality of first windows, second feature information included in the first feature information, includes: Determining, for each of the plurality of first windows, a similarity score for each of the first windows, the similarity score of the first window being determined based on whether the first window includes pixels related to a target object and a percentage of a window area in the first window occupied by the pixels related to the target object, the target object being an object of interest in the first remote sensing image; determining at least two third windows from the plurality of first windows according to the similarity score of each of the first windows, wherein the similarity scores of the at least two third windows are greater than the similarity scores of the remaining first windows in the plurality of first windows except for the at least two third windows; The second feature information included in the first feature information is determined based on the window aggregation features of the at least two third windows.

8. The method according to claim 5, characterized in that The deep feature extraction module of the image processing model includes a channel attention block, and the window attention integration block and the channel attention block are used to enable the deep feature extraction module in the image processing model to pay attention to both spatial local similarity and global channel correlation to avoid the limitations of a single attention mechanism; Determining, according to the window aggregation feature of each first window among the plurality of first windows, second feature information included in the first feature information, includes: performing channel attention calculation on the feature map corresponding to the first feature information by a channel attention block in a deep feature extraction module of the image processing model to determine the inter-channel correlation corresponding to the first feature information; The inter-channel correlation corresponding to the first feature information and the window aggregation features of the at least two third windows are integrated through an adaptive feature fusion layer to obtain the second feature information included in the first feature information.

9. The method according to claim 1, characterized in that The image processing model is generated in at least one of the following ways: Obtaining a training set of the image processing model, and training and updating the image processing model according to the training set of the image processing model, wherein the training set includes a fourth remote sensing image generated by performing bicubic downsampling on the third remote sensing image and adding noise, wherein the third remote sensing image is a remote sensing image having a resolution greater than a preset value; The image processing model is trained and updated according to a preset model training method; the model training method includes the following implementation process: in a first training stage, the image processing model is optimized by minimizing the pixel loss function L1 loss, and based on the configuration of the initial learning rate, the learning rate decays by 50% every 10,000 iterations; and, in a second training stage, the image processing model is fine-tuned by using the MSE function, and based on the configuration of the initial learning rate, the learning rate decays by 50% every 200,000 iterations.

10. A remote sensing image processing device, characterized in that: The device comprises: a shallow feature extraction module, configured to generate first feature information of the first remote sensing image using the shallow feature extraction module in the image processing model, wherein the shallow feature extraction module is configured with multiple depthwise separable convolutions used in parallel, each of the multiple depthwise separable convolutions having convolution kernels of different sizes, and the first feature information includes contour features and texture features extracted for objects of different sizes in the first remote sensing image using the convolution kernels of different sizes; a deep feature extraction module, configured to extract second feature information from the first feature information using the deep feature extraction module in the image processing model, wherein the second feature information is an abstract and semantic expression of the contour features and texture features represented by the first feature information; An image reconstruction module is used to reconstruct an image using the image reconstruction module in the image processing model according to the second feature information and the first remote sensing image to obtain a second remote sensing image.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the remote sensing image processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the remote sensing image processing method according to any one of claims 1 to 9 when executed.

13. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the remote sensing image processing method according to any one of claims 1 to 9.

Images