On-orbit satellite image super-resolution calculation method, system, equipment and medium

Abstract

The invention provides an in-orbit satellite image super-resolution calculation method and system, and the method comprises the steps: preprocessing low-resolution image data captured by a satellite platform through a data preprocessing module, and obtaining a group of low-resolution satellite image input; through a shallow feature extraction module, a shallow feature extraction convolutional network is used for operating satellite image input, and shallow features are obtained; through a depth feature extraction module, using a depth feature extraction network to extract deep features including high-frequency space information from the shallow features; and through a resolution reconstruction module, according to the obtained shallow layer features and deep layer features, using a feature fusion network to perform fusion processing on the shallow layer features and the deep layer features to obtain a reconstructed super-resolution satellite image. According to the method, hardware acceleration is carried out on an on-orbit SISR task based on a bottom hardware operator of scaling point multiplication attention, and a super-resolution task of a satellite on-orbit image is efficiently realized.

Description

Technical Field

[0001] The present invention relates to the field of satellite remote sensing application technology, and in particular to a method, system, device and medium for calculating super-resolution of on-orbit satellite images based on a scaled point multiplication attention operator. Background Art

[0002] Single image super-resolution (SISR) is the task of reconstructing a high-quality image from a lower-resolution version. Due to its wide range of applications, the pursuit of effective and sophisticated super-resolution algorithms has become a research hotspot in computer vision. Following fundamental research, CNN-based strategies have long dominated the super-resolution field. These strategies primarily utilize techniques such as residual learning or recursive learning to develop network architectures, significantly advancing the advancement of super-resolution models.

[0003] CNN-based networks have achieved remarkable success in terms of performance. However, the inductive bias of CNNs limits SISR models from capturing long-range dependencies. Their inherent limitations stem from the parameter-dependent receptive field scaling and kernel size of convolutional operators within different layers, which may ignore non-local spatial information in the image.

[0004] To overcome the limitations associated with CNN-based networks, researchers introduced Transformer-based SISR networks, leveraging the ability to model long-range dependencies to improve SISR performance. This approach significantly enhances the capabilities of traditional CNN-based models on various benchmarks. These subsequent studies have leveraged Transformer to innovate various network architectures specifically for super-resolution tasks, demonstrating the continued advancement of SISR technology by exploring new architectural innovations and techniques.

[0005] When using Transformer-based SISR models to perform inference on various datasets, a common phenomenon is that the intensity distribution of feature maps changes more significantly as the network depth increases. This indicates that the model has learned spatial information and attention strength. However, at the end of the network, the gradient descent often drops off sharply, shrinking to a smaller range. This phenomenon suggests that this sudden change may be accompanied by a loss of spatial information, indicating the presence of an information bottleneck.

[0006] The field believes that although the Transformer-based network architecture significantly expands the receptive field through the shift window attention mechanism to solve the problem of small receptive field in CNN, as the network depth increases, it is prone to gradient bottlenecks due to the loss of spatial information. This implicitly limits the performance and potential of the model. Summary of the Invention

[0007] In response to the above-mentioned deficiencies in the prior art, the present invention provides a method, system, device and medium for calculating super-resolution of on-orbit satellite images based on a scaled point product attention operator.

[0008] According to one aspect of the present invention, a method for calculating super-resolution of on-orbit satellite images is provided, comprising:

[0009] Preprocess the low-resolution image data captured by the satellite platform to obtain a set of low-resolution satellite image inputs Among them, H, W and C in are the height, width and number of channels of the image respectively;

[0010] Use the shallow feature extraction convolutional network Shadow_Conv(·) to input the satellite image I LQ Perform the operation to obtain the shallow feature F0∈R H×W×C :

[0011] F0=Shadow_Conv(I LQ )

[0012] Using the deep feature extraction network H DF (·) Extract deep features F containing high-frequency spatial information from shallow features F0 DF ∈R H×W×C :

[0013] F DF =H DF (F0)

[0014] According to the shallow feature F0 and deep feature F DF , using feature fusion network H rec (·), for shallow features F0 and deep features F DF Perform fusion processing to obtain reconstructed super-resolution satellite images

[0015] I SR =H rec (F0+F DF ).

[0016] Preferably, the shallow feature extraction convolutional network and / or the deep feature extraction network are constructed using the Transformer architecture and using the scaled point multiplication attention operator to replace the original attention calculation mechanism in the Transformer architecture. The replaced attention mechanism includes the following three steps: linear conversion of input features, scaled point multiplication attention operation and linear conversion of data transmission features.

[0017] According to another aspect of the present invention, a super-resolution computing system for on-orbit satellite images is provided, comprising:

[0018] A data preprocessing module, which is used to preprocess the low-resolution image data captured by the satellite platform to obtain a set of low-resolution satellite image inputs;

[0019] Shallow feature extraction module, which is used to operate satellite image input using shallow feature extraction convolutional network to obtain shallow features;

[0020] A deep feature extraction module, which is used to extract deep features containing high-frequency spatial information from shallow features using a deep feature extraction network;

[0021] The resolution reconstruction module is used to fuse the shallow features and deep features obtained using a feature fusion network to obtain the reconstructed super-resolution satellite image.

[0022] According to a third aspect of the present invention, a computing device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the device may be used to execute the method described above in the present invention, or to execute the system described above in the present invention.

[0023] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it can be used to execute the method described above in the present invention, or to run the system described above in the present invention.

[0024] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0025] In terms of scaling the dot product attention operator acceleration:

[0026] Existing Transformer-based algorithms typically involve complex linear transformations, multi-head attention calculations, and multi-layer feature fusion when calculating attention mechanisms. This consumes large amounts of computing resources and makes it difficult to adapt to the limited computing power of satellites on-orbit. By introducing a scaled dot-product attention operator, this paper simplifies traditional attention calculations into three core steps (linear transformation of input features, scaled dot-product attention operation, and linear transformation of output features), significantly reducing redundant calculations.

[0027] Furthermore, the present invention optimizes the parallel computing process of the attention operator, reduces memory usage and computational complexity, makes it more suitable for the parallel acceleration capabilities of satellite hardware, and improves computing efficiency.

[0028] Furthermore, the present invention optimizes operators based on the characteristics of FPGA or ASIC chips of the satellite on-orbit platform to achieve real-time processing with low latency and low power consumption, thereby enhancing resource adaptability.

[0029] Furthermore, the scaled point product attention adopted in the present invention retains the ability to capture global information through adaptive weight adjustment, avoids the performance degradation caused by traditional simplification methods, and maintains accuracy.

[0030] In terms of satellite-specific onboard super-resolution reconstruction computing hardware equipment:

[0031] Existing satellite image processing relies heavily on post-processing at ground stations, resulting in poor timeliness and high transmission costs. Traditional super-resolution algorithms (such as CNN) are limited by computing resources on satellite-based equipment and struggle to meet real-time requirements.

[0032] The present invention directly deploys the hardware-optimized Transformer architecture (such as the DRCT module) and the scaling point multiplication operator on the onboard equipment, reduces the dependence on data transmission, realizes "processing while shooting", and achieves real-time processing on orbit.

[0033] Aiming at the memory and power consumption limitations of onboard equipment, the present invention adopts dynamic batch segmentation (such as 3×3 batch segmentation) and lightweight embedding operations to reduce peak memory requirements and improve hardware adaptability.

[0034] The present invention combines the preprocessing process of radiation correction and geometric correction to enhance the algorithm's fault tolerance to satellite imaging noise and distortion, ensure stable operation in an on-orbit environment, and improve robustness.

[0035] In terms of multi-scale feature fusion of on-orbit satellites:

[0036] Due to the limitation of fixed receptive field, traditional CNN models have difficulty in fusing multi-scale spatial information, resulting in the loss of high-frequency details; while ordinary Transformers are prone to losing local features in deep networks due to information bottleneck problems.

[0037] The present invention uses a dense residual connection transformer (DRCT) and a residual deep feature extraction group (RDG), utilizes residual skip connections to reuse shallow features, and combines the shift window attention mechanism to dynamically capture global and local information, thereby enhancing multi-scale features.

[0038] The present invention introduces a twin network (STL) to realize cross-scale feature splicing, and balances the contribution of deep and shallow layer features through the scaling factor of the residual structure block (SDRCB), reduces the gradient vanishing problem, and realizes information flow optimization.

[0039] On satellite image datasets, the algorithm of the present invention has significantly enhanced detail recovery capabilities compared to existing ground super-resolution reconstruction models, especially in complex terrain and urban building scenes, thereby improving super-resolution quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0041] Figure 1 The figure is a workflow diagram of a method for calculating super-resolution of on-orbit satellite images in a preferred embodiment of the present invention.

[0042] Figure 2 Schematic diagram of the components of an on-orbit satellite image super-resolution calculation system in a preferred embodiment of the present invention.

[0043] Figure 3 Schematic diagram of an attention calculation mechanism optimized by using a scaled dot product attention operator in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention.

[0045] In order to implement the task of single image super-resolution (SISR) under the condition of limited on-orbit satellite resources, an embodiment of the present invention provides an on-orbit satellite image super-resolution calculation method. This method implements hardware acceleration for the on-orbit SISR task based on the underlying hardware operator of scaled point product attention, thereby efficiently achieving the super-resolution task of satellite on-orbit imagery.

[0046] Specifically, if Figure 1 As shown, the on-orbit satellite image super-resolution calculation method provided in this embodiment may include:

[0047] S1, pre-processing the low-resolution image data captured by the satellite platform to obtain a set of low-resolution satellite image input Among them, H, W and C in are the height, width and number of channels of the image respectively;

[0048] S2, using the shallow feature extraction convolutional network Shadow_Conv(·) to input the satellite image I LQ Perform the operation to obtain the shallow feature F0∈R H×W×C :

[0049] F0=Shadow_Conv(I LQ )(0.8)

[0050] S3, using deep feature extraction network H DF (·) Extract deep features F containing high-frequency spatial information from shallow features F0 DF ∈R H×W×C :

[0051] F DF =H DF (F0)(0.9)

[0052] S4, based on the shallow feature F0 and deep feature F DF , using feature fusion network H rec (·), for shallow features F0 and deep features F DF Perform fusion processing to obtain reconstructed super-resolution satellite images

[0053] I SR =H rec (F0+F DF ) (0.10).

[0054] In some preferred embodiments, the above S1, pre-processing the low-resolution image data captured by the satellite platform, may further include:

[0055] S11, parsing the low-resolution image data captured by the satellite platform, parsing the corresponding image data from the data field of the data frame, and in the case of multiple frames, performing a splicing operation on the image data to obtain the original image data;

[0056] S12, performing verification analysis on the original image data and cleaning abnormal data to obtain image data after preliminary quality screening;

[0057] S13, performing radiometric correction and geometric correction on the image data after preliminary quality screening, thereby obtaining a set of low-resolution satellite image inputs.

[0058] In some preferred embodiments, in the above method, the shallow feature extraction convolutional network and / or the deep feature extraction network are constructed using the Transformer architecture and using the scaled point multiplication attention operator to replace the original attention calculation mechanism in the Transformer architecture. The replaced attention mechanism includes the following three steps: linear conversion of input features, scaled point multiplication attention operation and linear conversion of data transmission features.

[0059] In some preferred embodiments, the above S2, shallow feature extraction convolutional network, uses 3×3 batch segmentation convolution Where P is the resolution of the corresponding segmented image batch, N is the number of image batches after segmentation, and N = HW / P 2 ;

[0060] The batch images after segmentation are subjected to batch embedding operation E, and then the position information and category flag are added to the embedded vector. Finally, the combined embedded vector is input into Transformer Encoder to obtain the shallow feature F0.

[0061] In some preferred embodiments, the above S3, deep feature extraction network, may further include: K residual deep feature extraction groups RDG and a single convolutional layer for feature conversion; wherein:

[0062] K residual deep feature extraction groups RDG process K intermediate features and then transform them through a single convolutional layer to obtain the final deep features, which are expressed as:

[0063] F i =RDG i (F i-1 ),i=1,2,…,K (0.11)

[0064] F DF =Conv(F K ) (0.12)

[0065] Where, F i is the i-th intermediate feature.

[0066] In some preferred embodiments, the above S3, the residual deep feature extraction group RDG, further uses the twin network STL to capture multi-scale attention information F j , expressed as:

[0067] F j =H trans (STL([F,…,F j-1 ])),j=1,2,3,4,5(0.13)

[0068] Where [·] represents multi-scale feature concatenation, H trans (·) represents a convolutional layer with nonlinear activation;

[0069] The residual depth extraction feature group RDG is further used to reuse the multi-scale attention information F using the residual structure block SDRCB j , used to enhance the receptive field of the deep feature extraction network, expressed as:

[0070] SDRCB(F j )=α·Z j+Z1 (0.14)

[0071] Where Z j is the multi-scale attention information F j The features obtained after nonlinear activation and batch normalization, α is the scaling factor of the residual structure.

[0072] Based on the same inventive concept, an embodiment of the present invention further provides an on-orbit satellite image super-resolution calculation system.

[0073] Specifically, if Figure 2 As shown, the on-orbit satellite image super-resolution computing system provided in this embodiment may include:

[0074] A data preprocessing module, which is used to preprocess the low-resolution image data captured by the satellite platform to obtain a set of low-resolution satellite image inputs;

[0075] Shallow feature extraction module, which is used to operate satellite image input using shallow feature extraction convolutional network to obtain shallow features;

[0076] A deep feature extraction module, which is used to extract deep features containing high-frequency spatial information from shallow features using a deep feature extraction network;

[0077] The resolution reconstruction module is used to fuse the shallow features and deep features obtained using a feature fusion network to obtain the reconstructed super-resolution satellite image.

[0078] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the method can be understood as preferred examples of constructing the system, which will not be elaborated here.

[0079] The technical solution provided by the above embodiment of the present invention is further described in detail below with reference to a specific application example.

[0080] In this specific application example, the on-orbit satellite image super-resolution calculation method involved is based on the scaled point product attention operator. It is aimed at on-orbit satellite platforms with limited computing resources, and realizes on-orbit super-resolution reconstruction that does not rely on ground post-processing, providing major assistance for satellite on-orbit detection and early warning tasks.

[0081] like Figure 1 and Figure 2 As shown in the figure, the super-resolution calculation method of on-orbit satellite images in this specific application example mainly includes the implementation of the following functional modules:

[0082] S1, data preprocessing module: Given a set of low-resolution satellite image inputs, represented as Among them, H, W and C in are the height, width and number of channels of the image respectively.

[0083] In S1, low-resolution image data captured by the satellite platform is obtained and preprocessed:

[0084] Before implementing super-resolution reconstruction of in-orbit satellite imagery, the data captured by the satellite platform must be parsed. Based on the hardware camera sensor's protocol, the corresponding image data is parsed from the data fields of the data frames. If multiple frames are involved, image stitching is also required. After obtaining the raw image data, preliminary verification and analysis are required to eliminate data with low signal-to-noise ratios.

[0085] For the image data after preliminary quality screening, basic radiation correction and geometric correction are implemented. Radiation correction reduces the radiation distortion caused by factors such as sensors, atmosphere and terrain; geometric correction mainly solves the geometric distortion caused by factors such as deformation of photographic materials, distortion of objective lenses, curvature of the earth, rotation of the earth, and terrain undulations.

[0086] S2, shallow feature extraction module: Use the shallow feature extraction convolution module Shadow_Conv(·) to operate on the input data to obtain the shallow feature F0∈R H×W×C :

[0087] F0=Shadow_Conv(I LQ )

[0088] In S2, the pre-processed image data is input and the shallow feature extraction module performs feature extraction:

[0089] According to the claims, the shallow extraction module is based on the Transformer architecture. Here we use the improved Vision Transformer (ViT) as an example, that is, using a 3×3 convolution module to replace a Transformer module in ViT to perform shallow fusion on the image. The Transformer module is represented as follows:

[0090] z l ′=MSA(LN(z l-1 ))+z k-1

[0091] z l =MLP(LN(z l ′))+z l '

[0092]

[0093] Among them, z l-1 is the feature input of the l-th layer Transformer module, LN(·) is a linear operation module for features, MSA(·) is a multi-head attention module, z l ′ is the feature output of the l-th layer multi-head attention module, MLP(·) is the multi-layer perceptron module, z l is the feature output of the l-th layer multi-layer perception module, and y is the output result of the ViT shallow feature extraction module.

[0094] Furthermore, the Transformer architecture is used in the shallow feature extraction module;

[0095] Considering that large batch segmentation of the input image (for example, the batch segmentation image size is 16×16) will cause the model to ignore a lot of detail information, resulting in poor generalization effect of the model effect, the present invention adopts 3×3 batch segmentation convolution Where P is the resolution of the corresponding segmented image batch, N = HW / P 2 N in represents the number of image batches after segmentation.

[0096] After segmentation, the batch images are subjected to batch embedding operation E, and then the position information and category flag are added to the embedded vector. Finally, the combined embedded vector is input into the Transformer module to obtain the shallow feature F0.

[0097] S3, deep feature extraction module: extracts deep features F containing high-frequency spatial information from F0 DF ∈R H×W×C , which can be expressed mathematically as follows:

[0098] F DF =H DF (F0)

[0099] Among them, H DF (·)-time deep feature extraction module, which contains K residual deep feature extraction groups (RDGs) and a single convolutional layer for feature transformation.

[0100] In S3, a deep feature extraction module is used to further extract deep features from shallow features:

[0101] In the architecture of the deep feature extraction module (DRCT), the number of residual deep feature extraction group (RDG) and residual structure block (SDRCB) units is set to 6, and the number of channels of the intermediate feature map is specified as 180. For the window-based multi-head self-attention module (W-MSA), the number of attention heads and window size are set to 6 and 16. In terms of data preparation, image patches of size 256×256 pixels are extracted from high-resolution images. To improve generalization ability, this specific embodiment applies random horizontal flipping and rotation enhancement.

[0102] Furthermore, the deep feature extraction module consists of K residual deep feature extraction groups (RDGs) and a single convolutional layer for feature transformation, where:

[0103] K residual deep feature extraction groups (RDGs) process K intermediate features and then use a single convolutional layer to transform them into the final deep features. The mathematical representation is as follows:

[0104] F i =RDG i (F i-1 ),i=1,2,…,K

[0105] F DF =Conv(F K )

[0106] In order to improve the information fusion of multi-scale features, RDG further adopts the twin network (STL) to capture multi-scale attention information, which is mathematically expressed as follows:

[0107] F j =H trans (STL([F,…,F j-1 ])),j=1,2,3,4,5

[0108] Among them, [·] represents multi-scale feature concatenation, H trans (·) denotes a convolutional layer with nonlinear activation.

[0109] In addition, in the residual depth extraction feature group (RDG), the residual structure block (SDRCB) is used to reuse features to enhance the receptive field of the model:

[0110] SDRCB(F j )=α·Z j +Z1

[0111] Among them, Z j is the multi-scale attention information F j The feature obtained after nonlinear activation and batch normalization, α represents the scaling factor of the residual structure, which is set to 0.2 here.

[0112] S4, resolution reconstruction module: Based on the shallow features and deep features obtained in S2 and S3, a feature fusion module is used to fuse the shallow features and deep features to obtain the reconstructed super-resolution satellite image. The following formula represents the fusion operation:

[0113] I SR =H rec (F0+F DF )

[0114] Among them, H rec (·) is the fusion of high-frequency deep features F DF And the module of shallow low-frequency feature F0.

[0115] In the above steps of this specific application example, both the shallow feature extraction module and the deep feature extraction module are based on the Transformer architecture. Considering the limited computing resources of on-orbit satellites, operators optimized for the Transformer architecture are used for parallel acceleration.

[0116] Based on satellite hardware resource considerations, the attention calculation mechanism in the original Transformer architecture is replaced by a scaled dot product attention operator. The replaced attention operator only contains three steps: linear transformation of input features, scaled dot product attention operation, and linear transformation of data features.

[0117] In the original attention calculation module, the scaling dot product attention operator optimizes the scaling operation, masking operation, and random inactivation operation, traversing all cases of these operations to avoid a large number of repeated calculations. The schematic diagram of the scaling dot product attention operator optimizing the attention module is shown in the figure below. Figure 3 Specifically, Figure 3 middle:

[0118] Figure 3 This paper demonstrates the core functional design of the proposed scaled dot-product attention operator optimized attention module, which aims to address the efficiency and resource bottlenecks of the traditional Transformer attention mechanism in satellite on-orbit computing scenarios. This module achieves efficient feature information fusion and accelerated processing by simplifying the computational process and optimizing hardware adaptability. The specific functional description is as follows:

[0119] 1. Linear transformation of input features

[0120] This module first applies a linear transformation to the input low-resolution satellite imagery features to generate query, key, and value matrices. Unlike traditional multi-head attention mechanisms, this one employs a single-head global attention design to avoid computational redundancy caused by multi-head splitting while retaining the ability to capture global spatial dependencies. This linear transformation directly adapts to the parallel computing units of satellite hardware (such as the matrix multiplication accelerator in an FPGA), reducing the need for intermediate feature storage.

[0121] 2. Scaling dot product attention operation

[0122] This step is the core optimization link and includes the following functions:

[0123] Dot product and scaling: The similarity between the query and the key is calculated through the dot product operation, and a scaling factor is introduced to stabilize the value range to avoid gradient explosion or vanishing.

[0124] Adaptive weight generation: The Softmax function is used to generate an attention weight matrix, dynamically adjusting the importance of different spatial locations and enhancing the ability to focus on high-frequency details in satellite imagery (such as building edges and terrain textures).

[0125] Hardware-level operation fusion: Parallelizes operations such as masking (such as ignoring occluded areas) and random deactivation (to prevent overfitting) with scaled dot product calculations at the hardware instruction level, eliminating the delay in branch judgment in traditional implementations and significantly improving computing efficiency.

[0126] 3. Output feature linear conversion

[0127] The feature matrix output by the scaled dot product attention is mapped to the final feature through a linear transformation and combined with a residual connection to ensure the effective fusion of shallow and deep features. This step directly connects to subsequent processing modules (such as residual blocks or multi-layer perceptrons), forming an end-to-end super-resolution reconstruction process.

[0128] An embodiment of the present invention further provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any one of the methods described in the foregoing embodiments of the present invention, or to run any one of the systems described in the foregoing embodiments of the present invention.

[0129] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. In addition, the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0130] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0131] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0132] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0133] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one location to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. Alternatively, the ASIC can be located in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0134] The on-orbit satellite image super-resolution calculation method, system, device and medium provided by the above-mentioned embodiments of the present invention simplify the traditional attention calculation into three core steps (input feature linear conversion, scaled point product attention operation, output feature linear conversion) by introducing the scaled point product attention operator, which significantly reduces redundant calculations. Furthermore, by optimizing the parallel computing process of the attention operator, the memory usage and computational complexity are reduced, making it more suitable for the parallel acceleration capability of satellite hardware, thereby improving computing efficiency. Furthermore, based on the FPGA or ASIC chip characteristics of the satellite on-orbit platform, the operator is optimized to achieve low-latency, low-power real-time processing, thereby enhancing resource adaptability. Furthermore, the scaled point product attention retains the ability to capture global information through adaptive weight adjustment, avoids the performance degradation caused by traditional simplification methods, and maintains accuracy.

[0135] The on-orbit satellite image super-resolution calculation method, system, device and medium provided by the above-mentioned embodiments of the present invention are directly deployed on the on-board equipment through the hardware-optimized Transformer architecture (such as the DRCT module) and the scaling point multiplication operator, thereby reducing the dependence on data transmission, realizing "processing while shooting", and achieving real-time processing on orbit. In view of the memory and power consumption limitations of the on-board equipment, dynamic batch segmentation (such as 3×3 batch segmentation) and lightweight embedding operations are adopted to reduce the peak memory demand and improve the hardware adaptability. The pre-processing process of combining radiation correction and geometric correction is used to enhance the algorithm's fault tolerance to satellite imaging noise and distortion, ensure stable operation in the on-orbit environment, and improve robustness.

[0136] The above-mentioned embodiments of the present invention provide an in-orbit satellite image super-resolution calculation method, system, device and medium. Through the dense residual connection transformer (DRCT) and the residual deep feature extraction group (RDG), the residual jump connection is used to reuse shallow features, and the shift window attention mechanism is combined to dynamically capture global and local information, thereby enhancing multi-scale features. The twin network (STL) is introduced to realize cross-scale feature splicing, and the scaling factor of the residual structure block (SDRCB) is used to balance the contribution of deep and shallow features, reduce the gradient vanishing problem, and realize information flow optimization. Compared with the existing ground super-resolution reconstruction model, the detail recovery capability is significantly enhanced, especially in complex terrain and urban building scenes, which improves the super-resolution quality.

[0137] Matters not mentioned in the above embodiments of the present invention are well known in the art.

[0138] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for calculating super-resolution of on-orbit satellite images, characterized in that: include: Preprocess the low-resolution image data captured by the satellite platform to obtain a set of low-resolution satellite image inputs Among them, H, W and C in are the height, width and number of channels of the image respectively; Use the shallow feature extraction convolutional network Shadow_Conv(·) to input the satellite image I LQ Perform the operation to obtain the shallow feature F0∈R H×W×C : F0=Shadow_Conv(I LQ ) (0.1) Using the deep feature extraction network H DF (·) Extract deep features F containing high-frequency spatial information from shallow features F0 DF ∈R H×W×C : F DF =H DF (F0) (0.2) According to the shallow feature F0 and deep feature F DF , using feature fusion network H rec (·), for shallow features F0 and deep features F DF Perform fusion processing to obtain reconstructed super-resolution satellite images I SR =H rec (F0+F DF ) (0.3).

2. The method for calculating super-resolution of on-orbit satellite images according to claim 1, wherein: The pre-processing of low-resolution image data captured by the satellite platform includes: Parse the low-resolution image data captured by the satellite platform, parse the corresponding image data from the data field of the data frame, and in the case of multiple frames, splice the image data to obtain the original image data; Performing verification analysis and abnormal data cleaning on the original image data to obtain image data after preliminary quality screening; Radiometric correction and geometric correction are performed on the image data after the preliminary quality screening to obtain a set of low-resolution satellite image inputs.

3. The method for calculating super-resolution of on-orbit satellite images according to claim 1, wherein: The shallow feature extraction convolutional network and / or the deep feature extraction network are constructed by adopting the Transformer architecture and using the scaled point multiplication attention operator to replace the original attention calculation mechanism in the Transformer architecture. The replaced attention mechanism includes the following three steps: linear conversion of input features, scaled point multiplication attention operation and linear conversion of data transmission features.

4. The method for calculating super-resolution of on-orbit satellite images according to claim 3, wherein: The shallow feature extraction convolutional network uses 3×3 batch segmentation convolution Where P is the resolution of the corresponding segmented image batch, N is the number of image batches after segmentation, and N = HW / P 2 ; The batch images after segmentation are subjected to batch embedding operation E, and then the position information and category flag are added to the embedded vector. Finally, the combined embedded vector is input into Transformer Encoder to obtain the shallow feature F0.

5. The method for calculating super-resolution of on-orbit satellite images according to claim 3, wherein: The deep feature extraction network includes: K residual deep feature extraction groups RDG and a single convolutional layer for feature conversion; wherein: The K residual deep feature extraction groups RDG process the K intermediate features and then transform them through the single convolutional layer to obtain the final deep features, which are expressed as: F i =RDG i (F i-1 ),i=1,2,…,K (0.4) F DF =Conv(F K ) (0.5) Where, F i is the i-th intermediate feature.

6. The method for calculating super-resolution of on-orbit satellite images according to claim 5, characterized in that: The residual deep feature extraction group RDG uses the Siamese network STL to capture multi-scale attention information F j , expressed as: F j =H trans (STL([F,…,F j-1 ])),j=1,2,3,4,5 (0.6) Where [·] represents multi-scale feature concatenation, H trans (·) represents a convolutional layer with nonlinear activation; The residual depth extraction feature group RDG uses the residual structure block SDRCB to reuse the multi-scale attention information F j , used to enhance the receptive field of the deep feature extraction network, expressed as: SDRCB(F j )=α·Z j +Z1 (0.7) Where Z j is the multi-scale attention information F j The features obtained after nonlinear activation and batch normalization, α is the scaling factor of the residual structure.

7. An on-orbit satellite image super-resolution computing system, characterized in that: include: A data preprocessing module, which is used to preprocess the low-resolution image data captured by the satellite platform to obtain a set of low-resolution satellite image inputs; Shallow feature extraction module, which is used to operate satellite image input using shallow feature extraction convolutional network to obtain shallow features; A deep feature extraction module, which is used to extract deep features containing high-frequency spatial information from shallow features using a deep feature extraction network; The resolution reconstruction module is used to fuse the shallow features and deep features obtained using a feature fusion network to obtain the reconstructed super-resolution satellite image.

8. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When executing the computer program, the processor can be used to perform the method according to any one of claims 1 to 6, or run the system according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 6, or to run the system according to claim 7.

Images