Method and system for rapid backhaul of off-shore remote sensing satellite data

Abstract

The application relates to the technical field of satellite telemetry data transmission, and provides a fast backhaul method and a fast backhaul system for overseas remote sensing satellite data, the fast backhaul method comprising the following steps: S1, acquiring overseas remote sensing satellite telemetry data; S2, carrying out fragmentation processing on the overseas remote sensing satellite telemetry data to form fragmented files and metadata; the metadata records the fragmentation condition; S3, successively uploading the fragmented files and the metadata to the cloud; and S4, receiving the fragmented files and the metadata from the cloud in the country, and combining multiple fragmented files into a complete file according to the metadata. The scheme can adapt to the characteristics of large-volume data, combine the advantages of cloud services, and is a new transmission method for breaking through the cross-border link restriction, improving data utilization, and reducing comprehensive cost.

Description

Technical Field

[0001] This invention relates to the field of satellite telemetry data transmission technology, and in particular to a method and system for rapid backhaul of remote sensing satellite data from overseas. Background Technology

[0002] In recent years, aerospace remote sensing technology has achieved leapfrog development and has become a core means of acquiring geospatial information. Its applications have rapidly expanded from traditional land surveying and meteorological monitoring to cross-regional scenarios such as cross-border infrastructure planning, polar scientific expeditions, and overseas disaster emergency response, becoming a key technology supporting global resource allocation and sustainable development.

[0003] The maturity of high-resolution and multi-payload technologies has driven an exponential increase in remote sensing data volume. A single scene (50km × 50km) from a 0.5-meter resolution optical satellite generates 30GB of data, more than 80 times that of a 10-meter resolution satellite. Synthetic Aperture Radar (SAR) satellites, due to their all-weather operation, often generate over 100GB of data per scene, reaching up to 500GB in high-resolution mode. More importantly, modern remote sensing has shifted from "single-scene observation" to "continuous monitoring." This "high-resolution + high-frequency" model transforms remote sensing data from a "scarce resource" into a "massive asset," but it also brings severe challenges: overseas observation data needs to be transmitted back within the satellite's transit window, and bandwidth fluctuations and high packet loss rates in cross-border links become bottlenecks; simultaneously, the storage, verification, and distribution of terabyte-level data place enormous pressure on traditional architectures, urgently requiring new technologies to overcome the dilemmas of "difficult acquisition, slow transmission, and low utilization efficiency."

[0004] Currently, the transmission and management of overseas remote sensing satellite data mainly rely on three types of technical solutions, each of which has significant limitations in practice:

[0005] The first type is a dedicated line direct transmission and full file transmission solution. Implementation: This solution receives satellite data from an overseas ground station and then transmits it directly to a domestic central node using a cross-border SDH (Synchronous Digital Hierarchy) dedicated line or MPLS VPN (Multiprotocol Label Switching Virtual Private Network). Data is transmitted as a complete file. A global MD5 hash value is generated before transmission, and the receiving end performs a consistency check after downloading the entire file. Limitations of this solution: 1. Dedicated line bandwidth is fixed (typically 100Mbps-1Gbps), resulting in high costs. It is also difficult to adapt to the sudden data transmission demands during satellite transit windows; 2. If a network interruption occurs during full file transmission, the entire file must be retransmitted, far exceeding the transmission window period; 3. MD5 verification requires waiting for the entire file to be received, making it impossible to detect transmission errors in real time.

[0006] The second type involves a self-built storage and distribution architecture. Implementation involves deploying an independent storage cluster (such as a distributed storage system based on GlusterFS) domestically, equipped with multiple physical servers as download nodes. Data storage uses a RAID5 / 6 redundancy mechanism, and users download via FTP or HTTP protocols. Limitations include: 1. Storage capacity expansion requires advance hardware procurement, which can easily lead to idle resources or insufficient capacity as the annual data volume increases; 2. Limited concurrent download capabilities, with single-node bandwidth ≤1Gbps, and the speed per user dropping below 10Mbps when 100 users download simultaneously; 3. Off-site disaster recovery requires additional deployment of backup clusters, increasing hardware and maintenance costs.

[0007] The third category is basic encryption and coarse-grained authorization mechanisms. Implementation: Data transmission uses SSL / TLS encryption, access control relies on simple account and password verification, and some systems restrict access through IP whitelists. Limitations: 1. SSL / TLS encryption only applies to the transport layer; data storage is mostly in plaintext, posing a risk of leakage; 2. Authorization granularity is coarse, unable to differentiate control based on data type or user role, and prone to data misuse.

[0008] It is evident that existing technologies suffer from the following four main drawbacks:

[0009] 1. Cross-border transmission is inefficient, limited by link conditions and time windows.

[0010] Remote sensing data from overseas needs to be received by overseas ground stations when the satellite passes over the country, and then transmitted back to the domestic processing center via cross-border links. Traditional transmission methods have two major limitations:

[0011] Time window constraints: The satellite's transit time over foreign ground stations is usually only 5-15 minutes. It is necessary to complete the reception and initial transmission of massive amounts of data in a short period of time, otherwise the opportunity to acquire data will be missed.

[0012] Unstable cross-border links: Existing cross-border public network links have large bandwidth fluctuations, high packet loss rates, and significant delays, resulting in insufficient data return completion rates for single cross-border transactions, and a large amount of data is discarded due to transmission timeouts.

[0013] 2. The integrity protection mechanism for large file transfers is outdated.

[0014] Traditional solutions employ a "full file transfer + end-to-end verification" model, which has significant drawbacks:

[0015] Weak fault tolerance: If the file is corrupted due to network interruption during transmission, the complete file needs to be retransmitted. Retransmission of large files can take several hours, far exceeding the satellite transit window.

[0016] Low verification efficiency: It relies on single hash verification algorithms such as MD5 and SHA-1, and can only perform integrity verification after the entire file has been received. It cannot detect fragmented data errors in real time, which leads to the problem being exposed in the later data processing stage, increasing rework costs.

[0017] 3. High storage and distribution costs and low resource utilization.

[0018] Existing technologies largely rely on self-built storage and distribution systems, which present the following problems:

[0019] Poor storage scalability: Remote sensing data is growing explosively, and building your own storage array requires advance planning of hardware investment, which can easily lead to extreme situations such as "resource idleness" or "insufficient capacity".

[0020] Low distribution efficiency: When multiple users download concurrently (such as in disaster emergency response scenarios), the bandwidth bottleneck of self-built servers is obvious, and the download speed of a single user is often less than 10Mbps;

[0021] High operation and maintenance costs: It requires the deployment of off-site disaster recovery, data backup and other infrastructure, which increases hardware maintenance and energy consumption costs.

[0022] 4. Inadequate data security and authorization management mechanisms.

[0023] Overseas remote sensing data often involves sensitive geographic information, and existing solutions have security vulnerabilities:

[0024] Lack of encryption during transmission: Some systems use plaintext transmission, which poses a risk of data interception and tampering;

[0025] Coarse-grained authorization: It often adopts a "one-size-fits-all" access control model, which cannot restrict the scope and permissions of data download based on user roles, and is prone to data abuse.

[0026] Therefore, existing technologies cannot meet the needs of "efficient transmission, secure storage, and accurate distribution" of overseas remote sensing satellite data. There is a need to provide a rapid backhaul method and system for overseas remote sensing satellite data that can adapt to the characteristics of large-volume data and combine the advantages of cloud services. This is a brand-new transmission method that breaks through the limitations of cross-border links, improves data utilization, and reduces overall costs.

[0027] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0028] The main purpose of this invention is to overcome the problem of not being able to meet the requirements of "efficient transmission, secure storage, and accurate distribution" of overseas remote sensing satellite data, and to provide a rapid backhaul method and system for overseas remote sensing satellite data. This method is adaptable to the characteristics of large-volume data and combines the advantages of cloud services. It is a brand-new transmission method that breaks through the limitations of cross-border links, improves data utilization, and reduces overall costs.

[0029] To achieve the above objectives, the first aspect of the present invention provides a method for rapid backhaul of overseas remote sensing satellite data, comprising the following steps:

[0030] S1: Acquire remote sensing satellite telemetry data from overseas;

[0031] S2: The remote sensing satellite telemetry data from overseas is fragmented to form fragment files and metadata; the metadata records the fragmentation details;

[0032] S3: Continuously upload fragmented files and metadata to the cloud;

[0033] S4: Receives fragmented files and metadata from the cloud within the country, and merges multiple fragmented files into a complete file based on the metadata.

[0034] According to an exemplary embodiment of the present invention, in step S1, acquiring remote sensing satellite telemetry data from overseas includes:

[0035] It connects directly to the baseband storage system via Fibre Channel;

[0036] Map binary files in baseband storage to local storage.

[0037] According to an exemplary embodiment of the present invention, step S2, acquiring remote sensing satellite telemetry data from overseas includes:

[0038] S21: Analyze the characteristics of telemetry data frames;

[0039] S22: Generate a frame index table based on the characteristics of telemetry data frames;

[0040] S23: Calculate the fragment size;

[0041] S24: Perform frame boundary alignment and segmentation based on the frame index table and fragment size;

[0042] S25: Create fragmented files and metadata.

[0043] According to an exemplary embodiment of the present invention, step S25 further includes: performing fragment encryption before forming the fragmented file; the fragment encryption information is recorded in the metadata.

[0044] According to an exemplary embodiment of the present invention, step S4 further includes: when receiving the fragmented file and metadata from the cloud within the territory, performing fragment decryption based on the metadata.

[0045] According to an exemplary embodiment of the present invention, the key for fragment decryption is obtained via the cloud.

[0046] According to an exemplary embodiment of the present invention, in step S22, generating a frame index table based on the characteristics of the telemetry data frame includes:

[0047] Based on the characteristics of telemetry data frames, the frame synchronization code in the telemetry data frames is detected by a sliding window.

[0048] Each time a frame synchronization code is detected, it is determined to be the start of a new frame;

[0049] The frame index table records the position information of each frame.

[0050] According to an exemplary embodiment of the present invention, in step S23, calculating the fragment size includes:

[0051] The size of the fragment is calculated based on the amount of telemetry data and the satellite transit window.

[0052] The fragment size is constrained by the minimum and maximum frame alignment units.

[0053] According to an exemplary embodiment of the present invention, step S24, the frame boundary alignment segmentation based on the index table and the fragment size includes:

[0054] Based on the frame index table, the gaps between frames are cut near the boundaries of the fragment size.

[0055] According to an exemplary embodiment of the present invention, in step S4, merging multiple fragmented files into a complete file based on metadata includes:

[0056] Create an empty file of the same size as the telemetry data from overseas telemetry satellites;

[0057] Write the data to an empty file sequentially based on the offset position corresponding to each fragment in the metadata.

[0058] The continuity of frames is checked in real time during the writing process.

[0059] As a second aspect of the present invention, the present invention provides a rapid backhaul system for overseas remote sensing satellite data, comprising: an overseas data acquisition module, a fragmentation module, a synchronous upload module, and a receiving and merging module;

[0060] The overseas data acquisition module is used to acquire overseas remote sensing satellite telemetry data;

[0061] The fragmentation module is used to fragment the remote sensing satellite telemetry data from overseas to form fragment files and metadata; the metadata records the fragmentation details.

[0062] The synchronous upload module is used to continuously upload unseen fragments and metadata to the cloud;

[0063] The receiving and merging module is used to receive fragmented files from the cloud within the territory and merge multiple fragmented files into a complete file based on metadata.

[0064] The advantages of this invention are:

[0065] This solution is adaptable to large-volume data and leverages the advantages of cloud services, offering a novel transmission method that overcomes cross-border link limitations, improves data utilization, and reduces overall costs. Specifically:

[0066] 1. Transmission efficiency is significantly improved.

[0067] The combination of dynamic fragmentation and parallel uploading greatly shortens the cross-border transmission time of remote sensing data, making it fully compatible with satellite transit windows.

[0068] 2. Data integrity and reliability assurance.

[0069] Encryption and verification mechanisms ensure zero data loss and zero tampering, avoiding rework in post-processing due to data corruption and reducing research and application costs.

[0070] 3. Optimize storage and maintenance costs.

[0071] Leveraging cloud-based elastic storage capabilities, capacity can be expanded on demand, avoiding the problem of idle resources in self-built storage and reducing storage costs; the integrated cloud-based architecture reduces hardware maintenance workload and significantly reduces manpower investment.

[0072] 4. Security compliance and controllable access.

[0073] The encryption and authorization mechanisms meet the security standards for cross-border data transmission, and the operation logs are traceable, effectively preventing data leakage and misuse.

[0074] 5. Improved user experience and application efficiency.

[0075] The receiver's multi-threaded download and automatic merging functions greatly shorten the download and processing time of files; the real-time push mechanism allows users to "process while downloading," improving the timeliness of data applications. Attached Figure Description

[0076] The above and other objects, features, and advantages of this application will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0077] Figure 1 The diagram illustrates the structure of a rapid data transmission system for overseas remote sensing satellites.

[0078] Figure 2 The diagram illustrates the steps of a rapid data transmission method for remote sensing satellite data from overseas. Detailed Implementation

[0079] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0080] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0081] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0082] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0083] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0084] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0085] This solution starts from the telemetry data source in baseband storage. First, it performs file sharing and fragmentation processing on the raw data, and simultaneously uploads the fragmented data to cloud object storage (such as Alibaba Cloud OSS). A dedicated file pull and push service is deployed in the cloud environment to pull the fragmented file stream from the cloud in real time and continuously store it on the MinIO file server. Then, the objects stored in Alibaba Cloud OSS are deleted to reduce the file storage cost of Alibaba Cloud OSS. Subsequently, authorized download clients push the fragmented files from the Minio file server in real time. After the client completes the download and verification of all fragmented files, it automatically performs merging processing to form a complete remote sensing satellite data file.

[0086] This solution, through a full-link design of "source fragmentation - cloud transfer - end-side aggregation," achieves efficient and secure transmission of telemetry data from baseband storage to the final client use. It ensures both the real-time nature of cross-border data transmission and data integrity through fragmentation verification and merging mechanisms. Details are as follows:

[0087] According to a first specific embodiment of the present invention, the present invention provides a rapid backhaul system for overseas remote sensing satellite data, such as... Figure 1 As shown, it includes: an overseas data acquisition module, a data sharding module, a synchronous upload module, and a receiving and merging module.

[0088] The overseas data acquisition module is used to acquire remote sensing satellite telemetry data from overseas.

[0089] The fragmentation module is used to fragment remote sensing satellite telemetry data from overseas into fragmented files and metadata; the metadata records the fragmentation details.

[0090] The synchronous upload module is used to upload fragmented files and metadata to the cloud in succession.

[0091] The receive and merge module is used to receive fragmented files from the cloud within the country and merge multiple fragmented files into a complete file based on metadata.

[0092] According to a second specific embodiment of the present invention, the present invention provides a method for rapid backhaul of overseas remote sensing satellite data, such as... Figure 2 As shown, it includes the following steps:

[0093] S1: Acquire remote sensing satellite telemetry data from overseas.

[0094] The overseas data acquisition module employs a high-speed data reading mechanism, deployed at an overseas ground station. The reading service uses a collaborative solution of "hardware interface + memory mapping" to achieve efficient acquisition of raw data.

[0095] Acquiring remote sensing satellite telemetry data from overseas includes:

[0096] S11: Directly connected to the baseband storage system via Fibre Channel.

[0097] It avoids the protocol overhead of traditional Ethernet by directly connecting to the baseband storage system via the Fibre Channel (FC) protocol (supporting 40Gbps link bandwidth).

[0098] S12: Maps binary files in baseband storage to local storage space.

[0099] By employing the mmap (memory-mapped file) mechanism, binary files in baseband storage are directly mapped to local storage space (instead of being copied to user space memory via read() / write() system calls), thereby increasing the data read rate to the 10GB / s level.

[0100] As a high-speed temporary storage medium, memory's write speed is highly matched with mmap's high-bandwidth read characteristics, reaching GB / s levels (such as the typical write speed of DDR5 memory, which is 4-8 GB / s), significantly outperforming disk write performance: HDD write speeds are approximately 100-200 MB / s, and ordinary SSDs only reach 500-1500 MB / s. Therefore, in scenarios requiring high-frequency read-write coordination, such as real-time baseband data processing and temporary cache file generation, this speed adaptability can completely eliminate the performance gap of "fast read, slow write," effectively ensuring end-to-end efficiency of data flow.

[0101] Performance advantages: By reducing the three-level data copying of "disk I / O - kernel buffer - user buffer" through memory mapping, the I / O overhead is reduced by more than 60%, avoiding data backlog during satellite transit windows caused by frequent disk reads and writes.

[0102] S2: The remote sensing satellite telemetry data from overseas is fragmented to form fragment files and metadata; the metadata records the fragmentation details.

[0103] Executed by the fragmentation module, the core function is to dynamically fragment raw telemetry data read from the baseband storage shared storage, solving the efficiency problem of cross-border transmission of large files.

[0104] Acquiring remote sensing satellite telemetry data from overseas includes:

[0105] S21: Analyze the characteristics of telemetry data frames.

[0106] The characteristics of telemetry data frames are:

[0107] The raw telemetry data uses the binary stream format of the CCSDS (Advisory Committee on Space Data Systems) standard, which has strict frame structure constraints:

[0108] Each frame has a fixed length of 4096 bytes (which can be dynamically configured according to the satellite payload type, but the frame length is uniform within the same batch of data), and consists of 4 parts:

[0109] Frame synchronization code, 4 bytes, fixed as 0x1ACFFC1D, used for frame boundary identification;

[0110] The main frame header, 20 bytes, contains metadata such as satellite ID, precise timestamp (millisecond level), and frame sequence number (continuously incrementing);

[0111] The payload data, 4064 bytes, is the raw optical / radar observation data and is the core content for subsequent processing;

[0112] The frame checksum, 8 bytes, uses the CRC-32 algorithm to verify the integrity of a single frame of data.

[0113] The data size of a single file ranges from 50GB to 2TB (depending on the satellite transit time and payload resolution; for example, a single transit of a high-orbit optical satellite can generate 1.2TB of data), and it consists of a continuous sequence of frames (e.g., 1.2TB of data contains approximately 3 million frames).

[0114] Key constraint: Data within a single frame cannot be divided. Therefore, if the fragmentation point falls in the middle of a frame (not at the start of the frame synchronization code), the synchronization code and check code of that frame will become invalid, resulting in "frame loss" or "data corruption" during parsing.

[0115] S22: Generate a frame index table based on the characteristics of telemetry data frames.

[0116] The frame index table generated based on the characteristics of telemetry data frames includes:

[0117] S221: Detect the frame synchronization code in the telemetry data frame using a sliding window based on the characteristics of the telemetry data frame;

[0118] S222: Each time a frame synchronization code is detected, it is determined to be the start of a new frame;

[0119] S223: The frame index table records the position information of each frame.

[0120] Specifically, during the data reading process, frame structure parsing is performed synchronously to generate a "frame index table" as the basis for fragment location:

[0121] Parsing logic: The reading service takes a byte stream as input and uses a sliding window (window size = frame synchronization code, length = 4 bytes) to detect the 0x1ACFFC1D sequence (i.e., frame synchronization code) in real time. Each time a frame synchronization code is detected, it is determined to be the start of a new frame.

[0122] Index table contents: Records key information for each frame in array form, including:

[0123] The frame start offset is the byte position relative to the file header, such as the first frame starting at byte 0 and the second frame starting at byte 4096.

[0124] The frame sequence number is taken from the frame header, and is continuous and non-repeating. It is used to verify the integrity of the frame sequence.

[0125] Timestamps, accurate to milliseconds, are used for tracing the time dimension of data;

[0126] The frame verification results are checked in advance using CRC-32, and invalid frames are marked to avoid wasting resources in subsequent processing.

[0127] Storage format: The index table is temporarily stored in memory in binary format (to avoid storage on disk I / O), and its size is about 0.5% of the original data (e.g., the index table of 1TB of data is about 5GB). It can be dynamically expanded as the data reading progresses.

[0128] S23: Calculate the fragment size.

[0129] S231: Calculate the fragment size based on the amount of telemetry data and the satellite transit window.

[0130] The formula for calculating the piece size based on the amount of telemetry data and the satellite transit window is as follows:

[0131]

[0132] Where S is the fragment size; B is the real-time bandwidth of the cross-border link in Mbps; T is the remaining transit time in seconds; η is the link utilization coefficient, a constant ranging from 0.7 to 0.9; and N is the preset maximum number of fragments.

[0133] S232: Constrain the fragment size based on the minimum frame alignment unit and the maximum frame alignment unit.

[0134] Constraints are added to the formula for calculating the fragment size to ensure that the fragment size is adapted to the frame structure.

[0135] The constraint is that the fragment size is less than or equal to the maximum frame alignment unit, and equal to the fragment size calculated by the basic formula, which is the maximum value between the fragment size and the minimum frame alignment unit.

[0136] Fragment size S = max("(B×T×η) / N", minimum frame alignment unit)

[0137] And S ≤ maximum frame alignment unit.

[0138] in:

[0139] (B×T×η) / N: The base fragment size calculated by the original formula (B is the real-time bandwidth of the cross-border link, in Mbps; T is the remaining transit time, in seconds; η is the link utilization coefficient, which is 0.7-0.9; N is the preset maximum number of fragments, which is recommended to be 200-500).

[0140] Minimum frame alignment unit: 16 × frame length (e.g., 16 × 4096 = 65536 bytes) to avoid metadata redundancy caused by excessively small fragments.

[0141] Maximum frame alignment unit: 1024 × frame length (e.g., 1024 × 4096 = 4MB), to avoid excessively large fragments leading to high retransmission costs.

[0142] S24: Perform frame boundary alignment and segmentation based on the frame index table and fragment size.

[0143] Frame boundary alignment and segmentation based on the index table and fragment size includes:

[0144] Based on the frame index table, the gaps between frames are cut near the boundaries of the fragment size.

[0145] Specifically, based on the "frame index table", the nearest "complete frame end position" (i.e. the end of the checksum of the last frame) is located near the boundary range of the calculated fragment size S, ensuring that the fragment cutting point falls in the gap between frames (no valid data area).

[0146] Example: If the calculated S is 3.8MB, and the end position of the most recent complete frame is 3.84MB (which contains exactly 938 complete frames: 938 × 4096 = 3,842,048 bytes), then the final size of the fragment file will be adjusted to 3.84MB.

[0147] S25: Create fragmented files and metadata.

[0148] Step S25 also includes: performing fragment encryption before forming the fragmented file; the fragment encryption information is recorded in the metadata.

[0149] Each fragment is encrypted using the AES-256-GCM algorithm (the key is dynamically generated by the cloud, such as Alibaba Cloud KMS, and associated with the satellite mission ID). After encryption, a 16-segment GCM authentication tag is appended to the end of the fragment (for on-side decryption verification).

[0150] GCM is a standardized process that "packages together the logic of encrypting and verifying the security of data".

[0151] After fragmentation, fragment metadata is automatically generated and packaged into a JSON-formatted index file. This metadata includes, but is not limited to: fragment sequence number, starting offset, SHA-256 hash value, total number of fragments, unique identifier of the original file, total size of the original file, ending offset, size of the encrypted fragment, number of complete frames, KMS key ID required for decryption, and overall hash value of the original file.

[0152] The metadata structure in JSON format is as follows:

[0153]

[0154]

[0155] After generating the metadata, the metadata is signed: the metadata is signed using the private key of the overseas ground station.

[0156] RSA signatures prevent tampering with public keys pre-installed in clients downloaded within the country.

[0157] RSA signatures are a digital signature technology based on the RSA public-key cryptosystem. They are primarily used to solve problems related to identity authentication, data integrity verification, and non-repudiation in the digital world, and are one of the foundational technologies in fields such as e-government, e-commerce, and cybersecurity.

[0158] S3: Continuously upload fragmented files and metadata to the cloud.

[0159] As a preferred implementation method, Alibaba Cloud OSS is selected for the cloud.

[0160] OSS (Object Storage Service).

[0161] OSS is a massive object storage service provided by cloud providers such as Alibaba Cloud and Amazon Web Services. Based on the object storage model (which is different from traditional file systems or block storage), it is specifically designed to store and manage unstructured data. Users can access the data anytime, anywhere through APIs, SDKs, or consoles.

[0162] During satellite transit, once the preceding fragmentation is completed, the data can be directly uploaded to the cloud by the synchronous upload module to improve transmission efficiency.

[0163] Call the Multipart Upload interface of Alibaba Cloud OSS Java SDK to upload fragmented data in parallel (concurrency = number of CPU cores × 2), and support breakpoint resume.

[0164] Metadata files are uploaded to the "Metadata Bucket" first (with access permissions set to "private + specified IP whitelist"), and fragmented data is uploaded to the "Data Bucket". The data is stored in directories according to file_id (i.e., the unique identifier of the original file). After the upload is completed, a "fragmentation ready" event is triggered to OSS.

[0165] On the cloud side, OSS shard file data is pulled in real time, transferred to MinIO, and the client is triggered to push and delete the pulled OSS shard files, realizing low-latency data transfer within the country.

[0166] MinIO is an open-source distributed storage system based on an object storage model. It is compatible with the Amazon S3 (SimpleStorage Service) API standard and can be deployed on private servers, public clouds, or hybrid cloud environments. It is specifically designed for storing massive amounts of unstructured data (such as files, images, videos, log streams, etc.).

[0167] 1. File retrieval service:

[0168] (1) Deploy the pull service on an Alibaba Cloud ECS instance (in the same region as OSS to reduce intranet latency) and use OSS event notifications to detect newly uploaded shard files in real time.

[0169] (2) Use the stream processing mode (InputStream) to pull the sharded data, avoid downloading the whole data to the local disk, and write it directly to the MinIO cluster through the MinIO SDK (configured as distributed mode, 3 replicas).

[0170] (3) After the OSS shard file is pulled, delete the OSS shard file. This reduces the cost of storing files on Alibaba Cloud OSS.

[0171] 2. MinIO Storage Management:

[0172] (1) Divide MinIO buckets according to data type, and store the fragmented files according to the original path (maintaining the same directory structure as OSS) to facilitate index association.

[0173] (2) Enable MinIO's version control function to record the modification history of the split files and prevent accidental deletion or overwriting.

[0174] (3) Set the storage time for cleaning up telemetry data files and perform regular cleaning to free up disk space occupied by Alibaba Cloud ECS instances.

[0175] 3. Push notification triggering mechanism:

[0176] (1) After the pull service completes the MinIO write of a shard, it automatically calls the push service API to add the shard metadata (including MinIO access address and hash value) to the "push queue" cache list.

[0177] S4: Receives fragmented files and metadata from the cloud within the country, and merges multiple fragmented files into a complete file based on the metadata.

[0178] This is executed by the receiving and merging module (i.e., the domestic client). Its core function is to authorize users to receive fragmented push notifications, verify their integrity, and then merge them into a complete file to ensure data consistency.

[0179] To enable authorized users to receive push notifications in segments, authorization and connection must first be established:

[0180] 1. The client logs in using an account and password plus a dynamic token (such as TOTP). After the server verifies the credentials, it returns a temporary access credential (including the MinIO access key and permission scope), which is valid for 2 hours.

[0181] A dynamic token is a time-limited, one-time authentication credential, unlike traditional static passwords (such as fixed passwords or API keys). It has a short validity period (usually a few seconds to a few minutes), and each token is different. It needs to be dynamically generated by combining a key with a specific algorithm (such as TOTP or HOTP) to improve the security of authentication.

[0182] 2. Establish a WebSocket long connection to receive real-time notifications of fragment availability from the push service.

[0183] WebSocket is a full-duplex communication protocol based on the TCP protocol, defined by the HTML5 standard. It addresses the limitations of the traditional HTTP protocol's request-response model (such as the need for clients to frequently poll to obtain real-time data), enabling persistent connections and bidirectional real-time communication between clients and servers.

[0184] Secondly, it is necessary to confirm whether the keys of the overseas and domestic terminals match, i.e., the acquisition and verification of the index file:

[0185] 1. The client requests the metadata file of the target file_id from the Alibaba Cloud OSS "Metadata Bucket", uses the preset public key to verify the RSA signature, and confirms that the metadata has not been tampered with.

[0186] 2. Parse the metadata to obtain information such as the total number of shards, the storage path, size, and hash value of each shard, and generate a "download task queue".

[0187] When receiving fragmented files and metadata from the cloud within the country, the fragments are decrypted based on the metadata.

[0188] 1. The client starts parallel download threads based on the number of local CPU cores (e.g., 8 threads for a 4-core CPU), downloads fragments in sequence according to the fragment number or allows out-of-order downloads, but the download status needs to be recorded.

[0189] 2. After each segment is downloaded: verify the SHA-256 hash value (compare with the metadata). If they do not match, mark it as "corrupted" and trigger a retransmission.

[0190] 3. The decryption key for the fragments is obtained from the cloud. Alibaba Cloud KMS is used to obtain the decryption key corresponding to kms_key_id (KMS key ID required for decryption). The decryption is performed using AES-256-GCM (the purpose is to verify the authentication tag and prevent the decryption process from being tampered with) to obtain the original fragment data.

[0191] Next, based on the metadata, the multiple fragmented files are merged into a complete file, including:

[0192] S41: Create an empty file of the same size as the telemetry data from overseas telemetry satellites.

[0193] The client creates an empty file the same size as the original file to pre-allocate disk space and reduce fragmentation.

[0194] S42: Write the data to an empty file in order of offset position corresponding to each fragment in the metadata.

[0195] Based on the start_offset and end_offset in the metadata, the original data of each fragment is written to the corresponding offset position. It should be written sequentially to avoid disk seek overhead.

[0196] S43: Real-time verification of frame continuity during the write process.

[0197] Real-time frame continuity verification during the merging process: Check whether the first and last frame numbers of adjacent fragments are consecutive by using the frame sequence number within the fragment (e.g., if the last frame number of fragment 1 is 1023, the first frame number of fragment 2 should be 1024). If they are not consecutive, mark it as "frame lost" and trigger the retransmission of the corresponding fragment.

[0198] After the merge is complete, verify the overall integrity:

[0199] 1. After merging, calculate the SHA-256 hash value of the complete file and compare it with the root_sha256 (the overall hash value of the original file) in the metadata. If they match, output "Merger successful" and delete the temporary fragment files.

[0200] 2. If there is an inconsistency, locate the difference range (by reverse-checking the fragment hash) and only retransmit the abnormal fragment to avoid re-downloading the entire file.

[0201] Enhance security and monitoring mechanisms throughout the entire transmission process:

[0202] 1. Data encryption:

[0203] Fragmented transmission uses TLS 1.3 encryption, and MinIO storage enables server-side encryption (SSE-KMS), with the key managed by Alibaba Cloud KMS.

[0204] 2. Monitoring Alarms:

[0205] Deploy Prometheus + Grafana to monitor link status (upload / download speed, fragment loss rate), and trigger alarms when transmission latency > 500ms or packet loss rate > 5%.

[0206] 3. Log auditing:

[0207] Record all upload, pull, and download operation logs for all segments (including timestamps, operators, and IP addresses).

[0208] The innovation of this solution lies in the following four aspects:

[0209] First, a dynamic adaptive fragmentation mechanism is designed in conjunction with cross-border transmission.

[0210] Breaking away from the traditional fixed-size fragmentation model, this paper proposes a dynamic fragmentation algorithm based on real-time bandwidth, satellite transit window, and link quality, achieving precise matching between fragment size and transmission capacity. Simultaneously, fragmentation processing is executed concurrently with Alibaba Cloud OSS upload, avoiding the sequential waiting of "fragmenting first, then uploading," thus improving cross-border transmission efficiency and resolving the transmission bottleneck of large files within a short time window.

[0211] Second, the cloud-native "pull-push" real-time workflow architecture.

[0212] An innovative collaborative storage mode between Alibaba Cloud OSS and MinIO is designed: domestic file retrieval services are triggered by OSS event notifications, and the data is directly written to MinIO in a stream processing manner (InputStream), avoiding the latency and storage overhead caused by full file persistence; deleting fragmented files retrieved from Alibaba Cloud OSS reduces Alibaba Cloud fragmented storage costs; at the same time, real-time push of fragmented files to the client is achieved based on WebSocket long connections, shortening the end-to-end latency of data uploaded from overseas to domestic clients, breaking through the traditional passive waiting mode of "client polling".

[0213] Third, a full-link data integrity verification and intelligent retransmission mechanism.

[0214] A two-layer verification system of "fragment-level hash verification + global hash verification" is constructed: each fragment generates an independent SHA-256 hash, which is then merged to calculate the global hash and compared with the index file to ensure zero data tampering; the integrity of fragments is verified in real time when the client downloads, and any anomalies are immediately triggered to trigger intelligent retransmission based on fragment granularity (only retransmitting the erroneous fragments), which saves redundant bandwidth compared to traditional full file retransmission.

[0215] Fourth, a layered security control and refined authorization system.

[0216] It integrates a three-tiered encryption mechanism—transmission layer, storage layer (MinIO SSE-KMS encryption), and application layer (Dynamic Token TOTP)—to address security vulnerabilities in cross-border data transmission and storage. Based on the RBAC model, it enables fine-grained control of user permissions (such as restricting download range by data type, region, and user role), and, in conjunction with operation log auditing, meets the compliance requirements for sensitive remote sensing data.

[0217] RBAC (Role-Based Access Control) model

[0218] RBAC is a mainstream access control model. Its core idea is to "associate users and permissions through roles"—instead of directly assigning permissions to users, it first defines "roles" (such as "administrator", "regular user", "researcher"), assigns corresponding permissions to roles, and then associates users with roles, thereby achieving batch management and flexible control of permissions.

[0219] Based on the aforementioned inventive points, this solution is adaptable to the characteristics of large-volume data and combines the advantages of cloud services, representing a novel transmission method that overcomes cross-border link limitations, improves data utilization, and reduces overall costs. Specifically:

[0220] First, transmission efficiency has been significantly improved.

[0221] The combination of dynamic fragmentation and parallel uploading significantly shortens the cross-border transmission time and satellite transit window reception time for remote sensing data. Simultaneously, uploaded files support Alibaba Cloud OSS global acceleration, achieving upload speeds of up to 50MB / s or more.

[0222] Second, data integrity and reliability assurance.

[0223] Dual-layer hash verification (individual verification and overall verification) and intelligent retransmission mechanism ensure zero data loss and zero tampering, avoiding rework in post-processing due to data corruption and reducing research and application costs.

[0224] Third, optimize storage and operation and maintenance costs.

[0225] Leveraging the elastic storage capabilities of Alibaba Cloud OSS and MinIO, storage can be expanded on demand, avoiding the problem of idle resources in self-built storage and reducing storage costs; the cloud-integrated architecture reduces hardware maintenance workload and significantly reduces manpower investment.

[0226] Fourth, security compliance and controllable access.

[0227] Three-level encryption (encryption between individual file fragments, complete files, and domestic and foreign clients) and a fine-grained authorization mechanism meet the security standards for cross-border data transmission. Operation logs are traceable, effectively preventing data leakage and misuse.

[0228] Fifth, improved user experience and application efficiency.

[0229] The client's multi-threaded download and automatic merging functions greatly shorten the download and processing time of files; the real-time push mechanism allows users to "process while downloading," improving the timeliness of data applications.

[0230] Exemplary embodiments of the present invention have been specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, arrangements, or implementations described herein; rather, the present invention is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for rapid backhaul of overseas remote sensing satellite data, characterized in that, Includes the following steps: S1: The overseas data acquisition module acquires overseas remote sensing satellite telemetry data from overseas ground stations during satellite transit; S2: The fragmentation module fragments the remote sensing satellite telemetry data from overseas into fragment files and metadata; the metadata records the fragmentation details. S3: Continuously upload fragmented files and metadata to the cloud; S4: Receives fragmented files and metadata from the cloud within the country, and merges multiple fragmented files into a complete file based on the metadata; In step S2, the remote sensing satellite telemetry data from overseas is processed into fragmented files, including: S21: Analyze the characteristics of telemetry data frames; S22: Generate a frame index table based on the characteristics of telemetry data frames; S23: Calculate the fragment size; S24: Perform frame boundary alignment and segmentation based on the frame index table and fragment size; S25: Create fragmented files and metadata.

2. The method for rapid backhaul of overseas remote sensing satellite data according to claim 1, characterized in that, In step S1, acquiring remote sensing satellite telemetry data from overseas includes: It connects directly to the baseband storage system via Fibre Channel; Map binary files in baseband storage to local storage.

3. The method for rapid backhaul of overseas remote sensing satellite data according to claim 1, characterized in that, Step S25 further includes: performing fragment encryption before forming the fragmented file; the fragment encryption information is recorded in the metadata; Step S4 further includes: when receiving the fragmented file and metadata from the cloud within the territory, performing fragment decryption based on the metadata.

4. The method for rapid backhaul of overseas remote sensing satellite data according to claim 3, characterized in that, The decryption key for each segment is obtained from the cloud.

5. The method for rapid backhaul of overseas remote sensing satellite data according to claim 1, characterized in that, In step S22, generating a frame index table based on the characteristics of the telemetry data frame includes: Based on the characteristics of telemetry data frames, the frame synchronization code in the telemetry data frames is detected by a sliding window. Each time a frame synchronization code is detected, it is determined to be the start of a new frame; The frame index table records the position information of each frame.

6. The method for rapid backhaul of overseas remote sensing satellite data according to claim 1, characterized in that, In step S23, calculating the fragment size includes: The size of the fragment is calculated based on the amount of telemetry data and the satellite transit window. The fragment size is constrained by the minimum and maximum frame alignment units.

7. The method for rapid backhaul of overseas remote sensing satellite data according to claim 1, characterized in that, In step S24, the frame boundary alignment segmentation based on the frame index table and the fragment size includes: Based on the frame index table, the gaps between frames are cut near the boundaries of the fragment size.

8. The method for rapid backhaul of overseas remote sensing satellite data according to claim 1, characterized in that, In step S4, merging multiple fragmented files into a complete file based on metadata includes: Create an empty file of the same size as the remote sensing satellite telemetry data from overseas; Write the data to an empty file sequentially based on the offset position corresponding to each fragment in the metadata. The continuity of frames is checked in real time during the writing process.

9. A rapid data transmission system for overseas remote sensing satellite data, characterized in that, include: Overseas data acquisition module, data segmentation module, synchronous upload module, and data receiving and merging module; The overseas data acquisition module is used to acquire overseas remote sensing satellite telemetry data through overseas ground stations during satellite transit. The fragmentation module is used to fragment the remote sensing satellite telemetry data from overseas to form fragmented files and metadata. The metadata records the fragmentation information; The process of fragmenting remote sensing satellite telemetry data from overseas into fragmented files includes: parsing the characteristics of telemetry data frames; generating a frame index table based on the characteristics of telemetry data frames; calculating the fragment size; performing frame boundary alignment and segmentation based on the frame index table and fragment size; and generating fragmented files and metadata. The synchronous upload module is used to upload the fragmented files and metadata to the cloud sequentially; The receiving and merging module is used to receive fragmented files from the cloud within the territory and merge multiple fragmented files into a complete file based on metadata.

Images