A multi-mode adaptive block compression method and system for high-entropy satellite signal data

Abstract

The application discloses a kind of multi-mode adaptive block compression methods and systems for high-entropy satellite signal data, belong to data compression and satellite communication technical field.The application is firstly scanned to original satellite signal data globally linearly, identifies and extracts all run-length encoding sections satisfying minimum length threshold;Then carry out arithmetic progression detection and independent coding to the remaining gap;Then the data not captured are adaptively blocked, and the optimal difference mode is selected from subtraction difference and XOR difference based on the highest byte zero rate fast detection to preprocess;Finally, rANS byte plane coding and difference bit packing coding are synchronously executed to each data block, and compete with original straight-through mode, select optimal coding result output.The application realizes the consideration of high compression ratio and high throughput under the premise of guaranteeing lossless compression by two-stage decoupling RLE extraction and block-level multi-mode competition architecture, and has strict zero-inflation guarantee.

Description

Technical Field

[0001] This invention relates to the fields of data compression and satellite communication technology, and in particular to a lossless compression technology for high-entropy satellite signal data. This invention is applicable to real-time lossless compression scenarios for raw sampled data of low-Earth orbit satellite signals from ground signal receiving stations. Background Technology

[0002] In recent years, low-Earth orbit (LEO) satellite communication, radar signal processing, and high-speed data acquisition systems have developed rapidly. Commercial space companies, particularly OneWeb and Starlink, have launched large-scale constellation deployment plans, driving the construction and operation of giant constellations in LEO. With the continuous increase in the number of satellites in orbit, the amount of satellite signal data that ground signal receiving stations need to process in real time is also increasing accordingly. Satellite signal data is crucial for spacecraft positioning, velocity measurement, and fault diagnosis, and its total volume is growing rapidly, placing enormous pressure on signal transmission channel bandwidth and storage resources. Therefore, achieving efficient compression of satellite signal data during transmission and storage is of significant practical importance.

[0003] Satellite signal data is characterized by high sampling rates, large data volumes, and wide numerical dimensions. The data is typically stored as 32-bit signed integers, with a broad numerical range and an overall high-entropy characteristic. Specifically, the actual acquired satellite signal data contains both periodic zero-value blocks (highly regular, continuously repeating patterns) and nearly randomly distributed high-entropy regions. The statistical characteristics of these two types of regions are drastically different, posing a significant challenge to the design of compression algorithms.

[0004] Existing general-purpose compression algorithms, such as gzip, LZMA, and Zstandard (zstd) based on the Deflate algorithm, have two inherent shortcomings when processing such high-entropy satellite signal data: (1) The general algorithm treats the input as an unstructured byte sequence and cannot make use of the correlation between int32 values. For example, the difference residuals between adjacent sampled values ​​are usually concentrated in the small value range, and this characteristic is not effectively utilized. (2) High-entropy regions account for a large proportion of satellite signal data. General algorithms have extremely poor coding performance in such regions, and a large amount of computing resources are wasted ineffectively, resulting in a significant decrease in compression throughput, making it difficult to meet the processing speed of hundreds of MB / s usually required by real-time acquisition systems.

[0005] Several studies have been conducted on satellite data compression, including lossless compression schemes using FPGA hardware platforms combined with LZW algorithms and RLE coding; fusion schemes that first perform LZW compression followed by Huffman coding; and compression schemes applying RLE to spacecraft packetized telemetry data. However, these studies primarily target spaceborne telemetry data, whose fixed frame length and low entropy characteristics differ fundamentally from the high entropy signal data from ground receiving stations. Therefore, the methods described are difficult to directly apply to the scenario addressed in this invention. Furthermore, existing schemes are mostly single-pass processing architectures. If a step-skip method is used when scanning data, some RLE segments may be missed, leading to a decrease in compression ratio.

[0006] In summary, there is currently a lack of a lossless compression scheme that can address the mixed characteristics of high-entropy satellite signal data while balancing high compression ratio and high real-time throughput. Summary of the Invention

[0007] The purpose of this invention is to propose a multi-mode adaptive block compression scheme for high-entropy satellite signal data. Under the premise of ensuring lossless compression, it can effectively balance high compression ratio and high throughput, meet the performance requirements of real-time processing of ground signal receiving stations, and overcome the problems of low compression throughput, inability to utilize numerical correlation, and large amount of invalid coding calculations in high-entropy regions of existing general compression algorithms when processing high-entropy satellite signal data.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: Solution 1: A multi-mode adaptive block compression method for high-entropy satellite signal data is provided, which includes the following steps: Step S1: Pattern extraction. Scan and identify the input satellite signal data and extract all run-length encoded segments that meet the minimum length threshold. Divide the original data into alternating run-length encoded segments and non-run-length encoded gap sequences. Step S2: Arithmetic sequence detection and encoding. Perform arithmetic sequence detection on each non-run-length encoding gap sequence generated in step S1, and encode and output the arithmetic segments that meet the minimum length threshold independently in the arithmetic sequence pattern. Step S3: Differential preprocessing and mode selection. The remaining data that was not captured by the arithmetic sequence detection in step S2 is divided into blocks of fixed block size, and differential preprocessing is performed on each data block. The optimal differential mode is selected from subtraction differential and XOR differential by a fast detection algorithm based on the highest byte zero rate. Step S4: Multi-mode competitive encoding. For each data block that has undergone differential preprocessing, rANS byte plane encoding and differential bit packing encoding are executed synchronously, and the output byte count is compared with the original pass-through mode. The mode with the smallest encoding result is selected as the optimal encoding scheme. Step S5: Output compressed file format. Write the global file header, block data area and block encoding sequence into the compressed file according to the multi-level compressed file format to form the final compressed output.

[0009] Furthermore, the method for identifying the run-length encoded segment in step S1 includes: starting from the current position of the input data, detecting whether adjacent element pairs are equal; after finding an equal pair, continuing to scan backward to determine the ending position and length N of the segment; if the segment length N is not less than the minimum threshold MIN_RLE_LEN, then the start and end positions of the segment are recorded in the ordered list rle_spans, and the segment is directly output in the run-length encoded format.

[0010] Further, the arithmetic sequence detection method in step S2 includes: calculating the common difference d = data[p+1] - data[p] from the current position p, and then verifying the arithmetic relationship element by element from p+2: whether data[j] - data[j-1] = d holds true. When the first position j that does not satisfy the equation is found, the interval [p, j] constitutes an arithmetic segment of length jp. If the length is not less than the threshold MIN_DELTA_LEN, it is encoded and output in the arithmetic sequence mode, and only the first term and the common difference are stored.

[0011] Furthermore, in step S3, the differential preprocessing includes the following two modes: Subtraction difference: r[i] = x[i] - x[i-1]; XOR difference: r[i] = x[i] ⊕ x[i-1]; The fast detection algorithm based on the highest byte zero rate uniformly samples 64 positions within the block, calculates whether the highest byte of the two difference results is zero, and selects the difference mode with the higher highest byte zero rate.

[0012] Furthermore, the rANS byte plane encoding described in step S4 includes the following sub-steps: (a) Byte plane separation: The residual sequence after difference is decomposed into several byte planes according to preset rules, and each plane is encoded independently; (b) Tri-state decision-making per plane: For each byte plane, constant detection, compression feasibility prediction and rANS encoding are performed in sequence; where compression feasibility prediction is performed by calculating Shannon entropy H through frequency statistics. If the sum of the estimated encoding size and the frequency table overhead is not less than the original size of the plane in bytes, the original data is stored directly and rANS encoding is skipped. (c) For the byte plane that passes the pre-detection, construct a frequency table and perform reverse rANS encoding.

[0013] Further, the differential bit packing encoding in step S4 includes: first performing ZigZag encoding on the differential residual sequence to map signed integers to unsigned integers; then scanning the entire sequence to determine the maximum value, thereby determining a fixed bit width b; and packing and storing each element in the sequence with a bit width of b.

[0014] Furthermore, the original pass-through mode described in step S4 includes continuous original block merging optimization. When multiple consecutive data blocks select the original pass-through mode, the consecutive original data blocks are merged into a single large block for output.

[0015] Furthermore, the multi-level compressed file format described in step S5 includes three levels: (a) Global file header: records the number of blocks, the original file size, and the size of each compressed block; (b) Block data area: Consists of N compressed sequence blocks arranged in sequence; (c) Intra-block coding sequence: Each coding block contains a fixed-length block header and a variable-length payload. The coding block types include run-length coding block, arithmetic sequence block, rANS block, differential bit packing block and raw pass-through block.

[0016] Option 2: A multi-mode adaptive block compression system for high-entropy satellite signal data, comprising: The pattern extraction module is configured to scan the input satellite signal data, identify and extract all run-length encoded segments that meet the minimum length threshold, and divide the original data into alternating run-length encoded segments and non-run-length encoded gap sequences. The arithmetic sequence detection and encoding module is configured to perform arithmetic sequence detection on each of the non-run-length encoded gap sequences, and independently encode and output the arithmetic segments that meet the minimum length threshold in an arithmetic sequence pattern; The differential preprocessing module is configured to divide the remaining data that was not captured by the arithmetic sequence detection into blocks of fixed block size, and perform differential preprocessing on each data block. The optimal differential mode is selected from subtraction differential and XOR differential by a fast detection algorithm based on the highest byte zero rate. The multi-mode competitive encoding module is configured to synchronously perform rANS byte-plane encoding and differential bit packing encoding on each data block that has undergone differential preprocessing, and compare the number of output bytes with the original pass-through mode, and select the mode with the smallest encoding result as the optimal encoding scheme. The file output module is configured to write the global file header, block data area and block encoding sequence into the compressed file according to the multi-level compressed file format to form the final compressed output.

[0017] Furthermore, the multi-mode contention coding module includes: rANS byte plane encoder, configured to perform byte plane separation, per-plane tri-state decision and reverse rANS encoding; Differential bit packing encoder, configured to perform ZigZag encoding, bit width determination and fixed bit width packing; The original pass-through encoder is configured to perform direct writing of raw data and optimization of merging consecutive raw blocks. The contention selector is configured to compare the number of output bytes of the three encoders and select the optimal encoding result.

[0018] Compared with the prior art, the solution provided by the present invention has the following beneficial effects: (1) High compression ratio and high throughput are balanced: Through a two-stage pipeline architecture and a multi-mode competition mechanism, the compression speed of this method is significantly better than gzip and zstd, and the decompression speed meets the real-time processing requirements of the ground receiving station.

[0019] (2) Completeness guarantee of run-length encoding pattern recognition: The two-stage decoupling design ensures that all run-length encoding segments that meet the threshold are completely recognized through independent global linear scanning, eliminating the problem of missing run-length encoding segments caused by step jumps in the traditional single-pass scheme.

[0020] (3) Strict zero inflation guarantee: The multi-mode competition mechanism uses Raw pass-through as a backup plan to ensure that the size of the compressed data does not exceed the size of the original data under any data distribution conditions.

[0021] (4) Fast processing of high-entropy regions: The plane-by-plane frequency pre-detection mechanism quickly identifies incompressible planes with low overhead before rANS encoding, avoiding the execution of invalid full encoding loops on high-entropy data, so that the system can maintain a high processing speed when high-entropy data is the main component.

[0022] (5) Strong adaptability: It can automatically identify run-length encoding patterns, arithmetic sequences, differentially compressible regions and high-entropy random regions in the data, and select the optimal encoding strategy for various local data features. Attached Figure Description

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0024] Figure 1 This is a diagram illustrating the overall architecture of the multi-mode adaptive block compression method for high-entropy satellite signal data of this invention, showing the data flow and interaction relationships between the modules in the two-stage pipeline.

[0025] Figure 2 This is a flowchart of the run-length encoding pattern extraction process in the multi-mode adaptive block compression method of the present invention, which shows the specific steps of the first stage of linear scanning to identify run-length encoded segments.

[0026] Figure 3 This is a flowchart of the block-level coding layer processing in the multi-mode adaptive block compression method of the present invention, which shows the processing of arithmetic sequence detection, differential preprocessing and competition of three coding paths in the second stage.

[0027] Figure 4 This is a diagram of the compressed file structure in the multi-mode adaptive block compression method of the present invention, showing the multi-level format of the global file header, block data area, and intra-block encoding sequence.

[0028] Figure 5 This is a diagram showing the overall system architecture of the multi-mode adaptive block compression system for high-entropy satellite signal data according to the present invention. Detailed Implementation

[0029] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0030] To address the technical problems of existing general-purpose compression algorithms when processing high-entropy satellite signal data, such as low compression throughput, inability to utilize the numerical correlation between adjacent sampling points, and wasted computational resources due to numerous invalid trial codes performed on high-entropy regions, this invention provides a multi-mode adaptive block compression method for high-entropy satellite signal data. This method organically integrates several innovative technical features with inherent synergistic relationships to form a dedicated lossless compression scheme for high-entropy mixed-characteristic data.

[0031] First, the present invention employs a two-stage decoupled compression processing architecture. The compression process is divided into a first stage (pattern extraction layer) and a second stage (block-level coding layer). The first stage, independent of subsequent block-level coding logic, performs a complete linear scan of the original data element-wise, identifying and extracting all run-length encoded segments that meet the length threshold, and immediately encoding and outputting them for complete removal from subsequent processing. The second stage only receives non-run-length encoded gap data, performing arithmetic sequence detection, differential preprocessing, and multi-pattern competitive coding. This architecture ensures that any repeating pattern meeting the threshold condition can be completely captured, eliminating the run-length encoded segment omission problem in traditional single-pass processing schemes.

[0032] Second, the present invention constructs a multi-mode adaptive competitive coding mechanism. Addressing the characteristic of interstitial data containing a mixture of arithmetic progressions, slowly varying signal regions, and high-entropy random regions, this scheme simultaneously executes three independent coding paths for each data block in the second stage: rANS byte-plane coding, differential bit packing coding, and raw pass-through. The optimal result is automatically selected by comparing the number of output bytes of the three paths. This mechanism allows the compression system to automatically adapt to the optimal coding strategy without prior knowledge of the data distribution characteristics.

[0033] Third, the present invention constructs an adaptive differential mode selection mode based on rapid detection of the highest byte zero rate. The present invention uniformly samples a finite number of positions within each data block, calculates whether the highest byte of the subtraction differential and XOR differential results is zero, and selects the differential mode with a higher highest byte zero rate. Accordingly, adaptive differential selection can be achieved with extremely low computational overhead, creating favorable numerical distribution conditions for subsequent byte-plane encoding.

[0034] Fourth, the present invention embeds a plane-by-plane frequency pre-detection mechanism in the rANS encoding path. On each byte plane, frequency statistics are first performed and the total encoding size (including frequency table overhead) is estimated. If the estimated value is not less than the original size of the plane, the original bytes are directly stored, skipping the complete reverse rANS encoding loop, effectively avoiding invalid encoding calculations on high-entropy planes.

[0035] Through the organic synergy of the above-mentioned technical features, the multi-mode adaptive block compression method for high-entropy satellite signal data provided by this invention can effectively balance high compression ratio and high throughput while ensuring lossless compression, and has a strict zero expansion guarantee.

[0036] The following combination Figure 1 The overall system architecture diagram shown is as follows: Figure 2 The flowchart for extracting run-length encoding patterns shown below and Figure 3 The block-level coding layer processing flowchart shown below provides a detailed explanation of the specific implementation scheme of the multi-mode adaptive block compression method for high-entropy satellite signal data provided by this invention.

[0037] Based on a two-stage decoupled compression processing architecture, the first stage of this method includes steps S1 to S3; the second stage includes steps S4 to S5.

[0038] Step S1: Pattern extraction (first stage).

[0039] refer to Figure 1 This step involves pattern extraction of the input satellite signal data. By performing a linear scan of the entire data segment, all run-length encoded (RLE) segments that meet the minimum length threshold are identified and extracted. The data is then divided into alternating RLE segments and non-RLE gaps. gap The RLE segment is defined as a subsequence of integers with consecutive equal values ​​and a length not less than MIN_RLE_LEN, and the start and end positions of all RLE segments are recorded in an ordered list rle_spans.

[0040] Step S2: Arithmetic sequence detection and encoding.

[0041] This step performs arithmetic progression (AP) detection on the non-RLE gap sequence obtained in step S1, and independently encodes and outputs arithmetic progression segments that meet the minimum length threshold in AP mode.

[0042] Step S3: Differential preprocessing and mode selection.

[0043] This step divides the remaining data that was not detected and captured by AP in step S2 into blocks of a fixed block size COMPRESS_BLOCK, and performs differential preprocessing on each data block. The optimal differential pattern is selected from subtraction differential and XOR differential by a fast detection algorithm based on the highest byte (MSB) zero rate.

[0044] Step S4: Multi-mode competition coding (second phase).

[0045] This step synchronously performs rANS byte-plane encoding (mode A) and differential bit packing encoding (mode B) on each differentially preprocessed data block, compares the size with the original raw data, and selects the mode with the smallest encoding result as the optimal encoding scheme for output.

[0046] Step S5: Output compressed file format.

[0047] This step writes the global file header, block data areas, and intra-block encoding sequences into the compressed file according to a multi-level compressed file format, forming the final compressed output.

[0048] The following section further elaborates on the specific implementation schemes for each step in the multi-mode adaptive block compression method.

[0049] See Figure 2 In step 1 of this multi-mode adaptive block compression method, when identifying and extracting run-length encoded (RLE) segments from the original satellite signal data (a continuous 32-bit signed integer sequence), a linear full scan is first performed on the input data, starting from the beginning of the data and detecting adjacent element pairs one by one. data [ i ], data [ iThe system checks if data[i] and data[i+1] are equal. When an equal pair is detected, the position is marked as the starting point of a candidate run-length encoded segment, and the scan continues until the first unequal element data[j] is encountered, thus determining a candidate segment interval [i, j) and its length N=ji. Then, the length N is compared with the preset minimum length threshold MIN_RLE_LEN: if the segment length N is not less than the minimum threshold MIN_RLE_LEN, the segment is determined as a valid run-length encoded segment, and its start and end positions are recorded in the ordered list rle_spans; if N is less than the threshold, it is not considered an independent run-length encoded segment, and the scan position only advances by one element. All recorded valid run-length encoded segments are directly output in the run-length encoding format, with the encoding form being [segment length (using Varint encoding), repeated integer value (fixed 4 bytes)]. The encoding cost of each segment is approximately 6 bytes, and compression benefits can be obtained when the repetition length N ≥ 2. After outputting, the scan position jumps directly to the end point j of the current run-length encoded segment, and subsequent detection continues. Through the above process, the original data is divided into alternating run-length encoded segments (RLE segments) and non-RLE gaps (...). gap )sequence: Data = [ gap 0][ RLE 1][ gap 1][ RLE 2]…[ gap k ][ RLE k ],in gap i Indicates the first i The RLE segment and the first i The non-repeating data area between +1 RLE segments. At the same time, all run-length encoded segments are removed from the data stream at this stage and do not proceed to the second stage of processing.

[0050] refer to Figure 3 In this multi-mode adaptive block compression method, step S2 involves processing each non-RLE gap sequence generated in step S1. gap i When performing arithmetic progression (AP) inspection at the gap level, the tolerance is calculated starting from the current position p of the gap. d = data [ p +1] - data [ p Then, starting from p+2, verify each element one by one. data [ j ]- data [ j -1] = dWhether it is valid or not.

[0051] When the first element that does not satisfy the equation is found at position j, the interval [ p , j This is considered an arithmetic progression segment, and its length is... j - p That is, the interval [ p , j The length of the structure is j - p Arithmetic progressions.

[0052] Next, the interval length j - p The length is compared with a preset threshold MIN_DELTA_LEN: if the length is not less than the threshold, the arithmetic sequence segment is independently encoded and output in Arithmetic Progression (AP) mode, with the encoding format being [segment length (Varint encoding), first term (4 bytes), common difference (4 bytes)]. Only the first term and common difference parameters need to be stored. Accordingly, all arithmetic sequences captured by the arithmetic progression mode are removed from the gap sequence and do not enter subsequent block processing.

[0053] As a supplementary explanation, the preset threshold MIN_DELTA_LEN is preferably set to 5, which can achieve the optimal balance between encoding benefits and avoiding data fragmentation.

[0054] In step S3 of this multi-mode adaptive block compression method, after dividing the remaining data not captured by the AP mode in step S2 into blocks, differential preprocessing is performed on each data block to reduce the numerical dynamic range. The differential preprocessing here employs the following two differential modes: Subtraction difference r [ i ] = x [ i ] - x [ i -1], applicable to numerically continuously smoothed sequences; XOR difference r [ i ] = x [ i ] ⊕ x [ i -1] is applicable to sequences with slow changes at high levels and fluctuations at low levels.

[0055] To avoid performing full differential encoding twice for each block, this step constructs a fast probing algorithm based on the highest significant byte (MSB) zero rate to make the optimal choice between the two.

[0056] Specifically, the fast detection algorithm based on the highest byte (MSB) zero rate is configured to uniformly sample 64 positions within the current data block, calculate the result for each sampling point under both differential modes, and count the proportion of samples where the highest byte (bits 31 to 24 of a 32-bit integer) is zero in both differential results. Based on this, the differential mode with the higher MSB zero rate is selected and applied to all data in the entire data block to generate a differential residual sequence. A higher MSB zero rate indicates that the differential values ​​are more concentrated in the lower bytes, which is beneficial for improving the compression ratio of the higher byte plane in subsequent byte plane encoding.

[0057] See Figure 3 In step S4 of this multi-mode adaptive block compression method, the following three encoding paths are executed synchronously for each differentially preprocessed data block (i.e., the differential residual sequence): Mode A (rANS byte plane encoding): rANS, a range variant of Asymmetric Numeral Systems (ANS), is an entropy coding method with coding efficiency approaching the lower bound of Shannon entropy. This scheme combines rANS with byte plane decomposition technology. First, byte plane decomposition is performed, decomposing each 32-bit integer in the differential residual sequence into four independent byte planes—P0 (lowest byte LSB), P1, P2, and P3 (highest byte MSB), with each plane encoded independently.

[0058] Next, a plane-by-plane three-state decision is executed independently for each plane: Constant check: Perform constant check on each byte plane. If all byte values ​​in the plane are the same, only store 1 byte of duplicate value. Compression feasibility is estimated by calculating Shannon entropy H through frequency statistics, estimating the sum of the coding size H×B / 8 and the frequency table overhead. If it is not less than the original plane size B bytes, the original bytes are stored directly and rANS coding is skipped to avoid invalid calculations on the high-entropy plane. rANS encoding: For planes that pass the pre-detection, construct a 12-bit precision frequency table and perform inverse rANS encoding.

[0059] The solution of this invention, based on the above-mentioned plane-by-plane frequency pre-detection mechanism, can guarantee high throughput in high-entropy scenarios.

[0060] Mode B (Differential Bitpacking): This mode constructs a fast encoding path for data blocks with narrow differential residual value ranges. First, ZigZag encoding is performed on the differential residual sequence: zigzag(x) = (x<<1) ⊕ (x>>31), mapping signed integers to unsigned integers. Then, the entire sequence is scanned to determine the maximum value, and from this, the required minimum fixed bit width b = ceil(log2(max_val+1)). Finally, each element in the sequence is packed and stored with a bit width of b, utilizing the dynamic range redundancy of the data to achieve efficient compression of data with narrow differential residual value ranges. This mode is suitable for scenarios such as slowly varying signals where differential residuals are concentrated in a small value range, achieving up to 16 times compression.

[0061] Mode C (Raw Through): When the data block byte distribution is nearly uniform and the residual value range after differential is wide, neither mode A nor mode B can produce compression benefits. In this case, the Raw pass-through mode is selected to write the original data directly to the output, with only an additional 7 bytes of block header overhead.

[0062] When multiple consecutive data blocks are selected in raw pass-through mode, continuous raw block merging is performed for output. That is, consecutive raw data blocks are merged into a single large block output, which can reduce the accumulated overhead of the block header in high-entropy regions. The accumulated overhead of the block header can be reduced from about 30 KB to about 35 bytes (a reduction of nearly three orders of magnitude).

[0063] Furthermore, the following optimal mode selection criteria are configured in this step: ; Among them | E ( x | indicates encoding E For data blocks x The number of bytes of the generated encoded output.

[0064] This mechanism provides a strict zero inflation guarantee: when neither rANS nor Bitpack can generate revenue, the system automatically selects Raw mode. Since Raw mode is always available, this mechanism provides a strict zero inflation guarantee, ensuring that the compressed data size does not exceed the original data size plus a finite block header overhead.

[0065] See Figure 4 In step S5 of this multi-mode adaptive block compression method, the compression result is written to a file according to a multi-level compressed file format, wherein the format includes three levels: Global file header: Located at the beginning of the compressed file, it records the number of chunks ChunkCount, the original file size OriginalSize, and the compressed size of each chunk ChunkSize. It uses Varint encoding and supports random location of any chunk during decompression. Block data area: Consists of N compressed sequence blocks arranged in sequence, and the byte length of each block is recorded by the corresponding ChunkSize in the header; Intra-block encoding sequence: Each encoding block contains a 17-byte fixed-length block header and a variable-length payload. Encoding block types include run-length encoded blocks (RLE blocks), arithmetic sequence blocks (AP blocks), rANS blocks, bitpack blocks, and raw blocks, arranged in the order of the original data position.

[0066] As further explanation, the satellite signal data targeted by this method is a continuous 32-bit signed integer sequence collected by a ground signal receiving station.

[0067] The multi-mode adaptive block compression method for high-entropy satellite signal data based on the above scheme has a compression speed of no less than 300 MB / s, a decompression speed of no less than 1600 MB / s, and the compressed data is no larger than the original data size under any circumstances.

[0068] The multi-mode adaptive block compression method for high-entropy satellite signal data presented in this example can be adapted into a corresponding software program to form a multi-mode adaptive block compression system for high-entropy satellite signal data. When running, this software program executes the steps of the multi-mode adaptive block compression method for high-entropy satellite signal data and stores them in a suitable storage medium for the processor to retrieve and execute.

[0069] See Figure 5 The resulting multi-mode adaptive block compression system for high-entropy satellite signal data mainly consists of the following modules: a pattern extraction module 100, an arithmetic sequence detection and encoding module 200, a differential preprocessing module 300, a multi-mode competitive encoding module 400, and a file output module 500. These modules are interconnected and work together to complete the conversion from raw data to compressed files.

[0070] The pattern extraction module 100 in the system is configured to receive a continuous 32-bit signed integer sequence as input and internally maintains a linear scan state machine to traverse the entire data sequence element by element.

[0071] Furthermore, the internal implementation of the pattern extraction module 100 includes the following logic: detecting whether adjacent element pairs are equal; when equality is detected, expanding backward to determine the boundary of the complete run-length encoded segment; comparing the segment length with the internally configured minimum length threshold MIN_RLE_LEN; for run-length encoded segments that reach the threshold, recording their start and end positions, and immediately converting the data into run-length encoded format ([length, repetition value]) and outputting it to the file output module 500; at the same time, marking these segments as processed from the original data stream, and only passing non-run-length encoded gap data to the downstream arithmetic sequence detection and encoding module 200.

[0072] The scanning process in the pattern extraction module 100 is independent of any subsequent block-level encoding logic, ensuring the integrity of run-length encoded segment recognition.

[0073] The arithmetic sequence detection and encoding module 200 in this system is configured to receive the gap data sequence from the pattern extraction module 100 and perform arithmetic sequence detection on it.

[0074] Furthermore, this module processes each gap sequence sequentially, starting from the beginning of the sequence, calculating the initial tolerance and verifying the persistence of the arithmetic progression element by element. When an arithmetic progression segment is detected, its length is compared with the internally configured threshold MIN_DELTA_LEN; if the threshold is reached, the segment is converted into an arithmetic sequence encoding format ([length, first term, tolerance]) and sent to the file output module 500; simultaneously, the segment is removed from the gap data. The remaining data not captured by the arithmetic progression detection is then output to the differential preprocessing module 300.

[0075] The differential preprocessing module 300 in this system is configured to divide the received remaining data into blocks according to a fixed block size COMPRESS_BLOCK, and perform adaptive differential mode selection for each data block.

[0076] Furthermore, the differential preprocessing module 300 includes a fast probing submodule. This fast probing submodule is configured to uniformly sample a preset number of positions (e.g., 64) from within the data block, calculate the results under both subtraction and XOR differential modes, and count the proportion of samples with the highest byte being zero in each result. Then, it selects the differential mode with the higher proportion and applies this mode to the entire data block to generate a differential residual sequence. Each data block (differential residual sequence) output by this module is packaged into a processing task and sent to the multi-mode competitive coding module 400.

[0077] The multi-mode contention coding module 400 in this system serves as the core processing unit, integrating three parallel-running sub-encoders: rANS byte plane encoder 410, differential bit packing encoder 420, and raw pass-through encoder 430, as well as a contention selector 440.

[0078] The multi-mode competitive coding module 400 receives each data block from the differential preprocessing module 300 and simultaneously starts three sub-encoders to independently encode the same data block.

[0079] The rANS encoder 410 internally implements complete logic for byte plane separation, plane-by-plane constant detection, compression feasibility prediction, and reverse rANS encoding.

[0080] The Bitpacking encoder 420 implements ZigZag transformation, global maximum value scanning, bit width determination, and fixed bit width packing.

[0081] The Raw Encoder 430 outputs raw data directly after only attaching the block header.

[0082] After the three sub-encoders complete their encoding, they each return the number of output bytes they generated to the contention selector 440.

[0083] The contention selector 440 interacts with the rANS byte plane encoder 410, the differential bit packing encoder 420, and the raw pass-through encoder 430, compares the size of the number of bytes output by the three, selects the encoding result with the smallest output as the final compressed representation of the data block, and sends it to the file output module 500.

[0084] Based on the above scheme, this multi-mode competitive coding module 400 also includes an optimization logic: detect continuous Raw coding output, and when the number of continuous Raw blocks reaches a preset threshold, merge the payloads of these Raw blocks and write only a merged block header during output.

[0085] The file output module 500 in this system is configured to assemble all the encoded blocks from the pattern extraction module 100, the arithmetic sequence detection module 200, and the multi-mode competition coding module 400 into a standard format compressed file according to the positional order of the original data.

[0086] Specifically, the file output module 500 first writes the global file header, which includes the total number of blocks, the total size of the original data, and the actual size of each block after compression (using Varint encoding). Then, it writes the block data areas in sequence. Each block consists of multiple variable-length encoded blocks arranged sequentially. Each encoded block consists of a 17-byte fixed-length block header (recording the encoding mode, the number of original elements, the payload length, and other information) and a variable-length payload that follows it.

[0087] The final output of this module is a self-describing, randomly accessible compressed file.

[0088] The resulting multi-mode adaptive block compression system for high-entropy satellite signal data adopts a two-stage decoupled compression processing architecture, and its operation process is as follows: Phase 1: Execution of the pattern extraction layer: After system startup, the pattern extraction module 100 performs a single linear scan on the input integer sequence. The scan begins at the start of the sequence and compares the values ​​of adjacent integers sequentially. When two adjacent integers are found to be equal, the module marks this position as the starting point of a candidate run-length encoded segment and continues scanning until the first unequal integer is encountered, thus determining the end point and length of the run-length encoded segment. The obtained length is compared with MIN_RLE_LEN (a value of 2). If the length reaches this threshold, the start and end positions of the run-length encoded segment are recorded in an ordered list rle_spans, and [length, repetition value] is output to the file output module 500 according to the run-length encoding format; the scan position then jumps directly to the end point of the segment and continues subsequent detection. If the length of the detected equal segment is less than the threshold, it is not considered a valid run-length encoded segment, and the scan position advances by only one element. After completing the full sequence scan, all identified run-length encoded segments have been removed from the original data, and the remaining non-run-length encoded gap sequences are sequentially sent to the arithmetic sequence detection and encoding module 200.

[0089] Second stage: Block-level coding layer execution: For each gap sequence, the arithmetic sequence detection and encoding module 200 starts from the beginning of the sequence, calculates the difference between the current element and the next element as a candidate common difference, and then verifies step by step whether subsequent elements continuously satisfy the arithmetic relationship. Once the verification fails at a certain position j, the interval [p, j) is determined as an arithmetic segment. The length of this arithmetic segment is compared with MIN_DELTA_LEN (value 5). If the length reaches the threshold, the arithmetic segment is output to the file output module 500 in the arithmetic sequence encoding format [length, first term, common difference]; at the same time, the segment is removed from the gap sequence. The remaining data not captured by the arithmetic sequence detection is passed to the differential preprocessing module 300.

[0090] After the above two steps, the remaining data is divided into blocks according to COMPRESS_BLOCK = 16384 bytes. For each data block, the differential preprocessing module 300 performs adaptive differential mode selection: 64 positions are uniformly sampled from the data block (sampling step size is 16384 / 64=256 bytes), the results of subtraction difference and XOR difference are calculated for each sampling point, the number of sampling points with the highest byte being zero in the two differential results is counted, the corresponding highest byte zero rate is calculated, and the one with the higher rate is selected as the optimal differential mode for the data block, and applied to all data in the block to generate a differential residual sequence.

[0091] Each differential residual sequence data block is fed into the multi-mode contention coding module 400. This module simultaneously starts three encoders: The rANS byte-plane encoder 410 decomposes each 32-bit integer in the data block into four byte planes. For each plane, it first checks whether it is a constant plane; if so, it stores only one repeating byte. If not, it performs frequency statistics on the plane, calculates the Shannon entropy H, and estimates the total size after rANS encoding (H×B / 8 + frequency table overhead). If the estimated value is not less than the original size B of the plane, it directly stores the original bytes of the plane, skipping the reverse rANS encoding; if the estimated value is less than B, it constructs a 12-bit precision frequency table and performs reverse rANS encoding.

[0092] The differential bit packing encoder 420 first performs a ZigZag transform on the differential residual sequence to map signed integers to unsigned integers; then it scans the entire sequence to determine the maximum value and the minimum fixed bit width b required; finally, it packs and stores each element in the sequence according to a bit width b.

[0093] The original pass-through encoder 430 simply appends a block header before the payload of the data block without performing any compression transformation.

[0094] After the three encoders have completed their parallel execution, the contention selector 440 compares the number of output bytes of the three and selects the smallest encoding result as the final compressed representation of the data block. If multiple consecutive data blocks are selected as Raw encoding, these Raw blocks are merged into a single large block output, retaining only the header of the merged block.

[0095] After all data blocks have been processed, the file output module 500 organizes the final output according to a multi-level compressed file format. First, a global file header is written, recording the total number of blocks, the total size of the original data, and the actual size of each compressed block (using Varint encoding). Then, the data areas of each block are written sequentially. Within each block, the encoded blocks are arranged in the order of the original data, and each encoded block consists of a 17-byte fixed-length header and a variable-length payload. This ultimately forms the compressed file.

[0096] The following example illustrates the application and implementation process of this invention in detail, using the real-time compression scenario of raw sampled data of low-orbit satellite signals from a ground receiving station as an example. This includes the system's structure and deployment, as well as the detailed process of the implementation method.

[0097] Example 1: Overall System Implementation and Compression Process This embodiment uses 72 MB of low-orbit satellite signal data (hexadecimal signal data, formatted as a continuous 32-bit signed integer sequence) collected by a ground receiving station as the processing object to fully illustrate the implementation of the method of the present invention.

[0098] Data characteristic analysis and parameter setting; The byte frequency distribution of actual satellite signal data exhibits a Laplace-like pattern: the 0x00 byte appears most frequently (approximately 2.58%), and the overall distribution decreases approximately symmetrically towards both sides of 0x00. There are approximately 240 periodic zero-byte segments in the data, with lengths concentrated in two formats: 4094 bytes and 4098 bytes, accounting for about 1.39% of the total file size. The non-zero region values ​​range from [-96457678, 619506027], showing a near-uniform random distribution.

[0099] Based on the above data characteristics, the system parameters are set as follows: MIN_RLE_LEN = 2 (minimum length threshold for RLE); MIN_DELTA_LEN = 5 (minimum length threshold for arithmetic progression); COMPRESS_BLOCK = 16384 (fixed block size for block-level coding, in bytes); rANS frequency table precision = 12 bits; MSB zero-rate sampling points = 64.

[0100] Phase 1: Execution of the pattern extraction layer.

[0101] The system performs a single linear scan on the input integer sequence: from position i = Starting from 0, check element by element data [ i ]and data [ i +1] Check for equality. When an equal pair is found, continue scanning forward until the first unequal position is reached. j Determine the RLE segment [ i , j ) length N = j - i .like N When ≥ MIN_RLE_LEN, then ( i , j Record it in the rle_spans list.

[0102] After the scan is complete, the data is segmented into RLE segments and RLE segments based on rle_spans. gap The sequence, RLE segment is immediately written to the output buffer in run-length encoded format.

[0103] For 72 MB of test data, this stage identified approximately 240 RLE segments, effectively processing about 1.39% of the data, with a corresponding area compression ratio of several hundred times.

[0104] Second stage: Block-level coding layer execution.

[0105] For each gap sequence gap i First, perform AP detection: from gap Starting position p Begin by calculating tolerances. d = data [ p +1] - data [ p ], verify the arithmetic progression element by element.

[0106] When the length of the arithmetic progression segment is greater than or equal to MIN_DELTA_LEN, the output is written in AP format, resulting in a net compression gain of approximately 45%. The remaining data after AP detection is divided into blocks of COMPRESS_BLOCK = 16384 bytes. For each block: Perform fast MSB zero-rate detection, sample 64 points, compare the MSB zero rate of subtraction difference and XOR difference, and select the higher one; Simultaneously initiate rANS byte plane encoding and Bitpack encoding, using Raw as the baseline; Select the scheme with the smallest encoded output from the three and write it into the block encoding sequence.

[0107] Frequency pre-detection mechanism accelerates high-entropy processing: For each byte plane in the rANS encoding, a single frequency count is first performed to calculate the Shannon entropy. H_est .like H_ est × B / 8 + Frequency table overhead ≥ B， in B If the number of bytes in the plane is less than the number of bytes in the plane, then the plane is determined to be incompressible, the complete rANS encoding cycle is skipped, and the original bytes are stored directly. In actual testing, data in high-entropy regions... P 2 、P The 3-byte plane almost entirely triggers this fast skip path, reducing the encoding latency in high-entropy regions to near the level of raw passthrough, ensuring that the system compression speed is always maintained above 200 MB / s.

[0108] Experimental results: On a 72 MB real satellite signal dataset, the comparison results with mainstream general compression algorithms are as follows: The compression speed of the method of this invention exceeds 300 MB / s (gzip is about 30 MB / s, zstd is about 40 MB / s), and the compression ratio is about 1.01; the decompression speed exceeds 1600 MB / s, which meets the requirements of real-time processing.

[0109] Example 2: Decompression Process This embodiment describes the decompression process corresponding to the compression method described above. The decompression process is performed according to the following steps: (1) Read the compressed file. First, parse the global file header to obtain the number of blocks (ChunkCount) and the original file size (OriginalSize). Based on the compressed size (ChunkSize) of each block recorded in the header (using Varint encoding), calculate the starting byte offset of each block in the file, thus supporting random access to any block.

[0110] (2) Read each block of data in sequence. For each block, parse the sequence of encoded blocks inside it in sequence. For each encoded block, first read the 17-byte fixed-length block header, and parse out the encoding mode (mode field, occupying 3 bits), the number of original elements (cnt field), and the payload size (sz field).

[0111] (3) Perform the corresponding decompression operation according to the encoding mode field: If the mode is run-length encoding: read the repetition values ​​and repetition counts from the payload, expand them according to the counts to generate the original integer sequence.

[0112] If the pattern is an arithmetic sequence: read the first term and the common difference in the payload, and restore the complete arithmetic sequence according to the segment length.

[0113] If the mode is rANS: First, read the frequency table, perform reverse rANS decoding to recover the byte sequence of each byte plane, then reassemble the four byte planes into a 32-bit integer sequence, and finally perform inverse differential transformation (if subtraction differential is used during compression, the inverse transformation is a prefix sum; if XOR differential is used, the inverse transformation is a cumulative XOR).

[0114] If the mode is differential bit packing: read the fixed bit width b, unpack each unsigned integer from the bit stream in sequence, and then perform the inverse ZigZag transform: x = (z >> 1) ⊕ (-(z & 1)), recover the differential residual sequence, and finally perform the inverse differential transform.

[0115] If the mode is Raw: the payload is copied directly to the output buffer.

[0116] (4) According to the original data position order of the encoded block within the block, the decoded data segments are spliced ​​together to restore the complete block data sequence. All blocks are spliced ​​together in order and verified against the original file size OriginalSize to confirm lossless restoration.

[0117] Throughout the decompression process, since each block is decoded independently and the Raw mode and differential bit packing mode are the main decompression paths, there is no need for complex entropy decoding dependencies, and the decompression speed can reach more than 1000 MB / s, which can meet the real-time processing requirements.

[0118] In response to the aforementioned multi-mode adaptive block compression scheme for high-entropy satellite signal data, this embodiment of the invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the steps of the aforementioned multi-mode adaptive block compression method.

[0119] This invention also provides a processor for running a program, wherein the program executes the steps of the above-described multi-mode adaptive block compression method during runtime.

[0120] This invention also provides a terminal device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. The program code is loaded and executed by the processor to implement the steps of the above-described multi-mode adaptive block compression method.

[0121] The present invention also provides a computer program product, which, when executed on a data processing device, is adapted to perform the steps of the above-described multi-mode adaptive block compression method.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0123] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A multi-mode adaptive block compression method for high-entropy satellite signal data, characterized in that, Includes the following steps: Step S1: Pattern extraction. Scan and identify the input satellite signal data and extract all run-length encoded segments that meet the minimum length threshold. Divide the original data into alternating run-length encoded segments and non-run-length encoded gap sequences. Step S2: Arithmetic sequence detection and encoding. Perform arithmetic sequence detection on each non-run-length encoding gap sequence generated in step S1, and encode and output the arithmetic segments that meet the minimum length threshold independently in the arithmetic sequence pattern. Step S3: Differential preprocessing and mode selection. The remaining data that was not captured by the arithmetic sequence detection in step S2 is divided into blocks of fixed block size, and differential preprocessing is performed on each data block. The optimal differential mode is selected from subtraction differential and XOR differential by a fast detection algorithm based on the highest byte zero rate. Step S4: Multi-mode competitive encoding. For each data block that has undergone differential preprocessing, rANS byte plane encoding and differential bit packing encoding are executed synchronously, and the output byte count is compared with the original pass-through mode. The mode with the smallest encoding result is selected as the optimal encoding scheme. Step S5: Output compressed file format. Write the global file header, block data area and block encoding sequence into the compressed file according to the multi-level compressed file format to form the final compressed output.

2. The method according to claim 1, characterized in that, The method for identifying run-length encoded segments in step S1 includes: starting from the current position of the input data, checking whether adjacent element pairs are equal; after finding an equal pair, continuing to scan backward to determine the ending position and length N of the segment; if the segment length N is not less than the minimum threshold MIN_RLE_LEN, then the start and end positions of the segment are recorded in the ordered list rle_spans, and the segment is directly output in run-length encoded format.

3. The method according to claim 1, characterized in that, The arithmetic sequence detection method in step S2 includes: calculating the common difference d = data[p+1] - data[p] from the current position p, and then verifying the arithmetic relationship element by element from p+2: whether data[j] - data[j-1] = d holds true. When the first position j that does not satisfy the equation is found, the interval [p,j] constitutes an arithmetic segment of length jp. If the length is not less than the threshold MIN_DELTA_LEN, it is encoded and output in the arithmetic sequence mode, and only the first term and the common difference are stored.

4. The method according to claim 1, characterized in that, In step S3, the differential preprocessing includes the following two modes: Subtraction difference: r[i] = x[i] - x[i-1]; XOR difference: r[i] = x[i] ⊕ x[i-1]; The fast detection algorithm based on the highest byte zero rate uniformly samples 64 positions within the block, calculates whether the highest byte of the two difference results is zero, and selects the difference mode with the higher highest byte zero rate.

5. The method according to claim 1, characterized in that, The rANS byte plane encoding described in step S4 includes the following sub-steps: (a) Byte plane separation: The residual sequence after difference is decomposed into several byte planes according to preset rules, and each plane is encoded independently; (b) Tri-state decision-making per plane: For each byte plane, constant detection, compression feasibility prediction and rANS encoding are performed in sequence; where compression feasibility prediction is performed by calculating Shannon entropy H through frequency statistics. If the sum of the estimated encoding size and the frequency table overhead is not less than the original size of the plane in bytes, the original data is stored directly and rANS encoding is skipped. (c) For the byte plane that passes the pre-detection, construct a frequency table and perform reverse rANS encoding.

6. The method according to claim 1, characterized in that, The differential bit packing encoding in step S4 includes: first performing ZigZag encoding on the differential residual sequence to map signed integers to unsigned integers; then scanning the entire sequence to determine the maximum value, thereby determining a fixed bit width b; and packing and storing each element in the sequence with a bit width b.

7. The method according to claim 1, characterized in that, The original pass-through mode mentioned in step S4 includes continuous original block merging optimization. When multiple consecutive data blocks select the original pass-through mode, the consecutive original data blocks are merged into a single large block for output.

8. The method according to claim 1, characterized in that, The multi-level compressed file format described in step S5 includes three levels: (a) Global file header: records the number of blocks, the original file size, and the compressed size of each block; (b) Block data area: Consists of N compressed sequence blocks arranged in sequence; (c) Intra-block coding sequence: Each coding block contains a fixed-length block header and a variable-length payload. The coding block types include run-length coding block, arithmetic sequence block, rANS block, differential bit packing block and raw pass-through block.

9. A multi-mode adaptive block compression system for high-entropy satellite signal data, characterized in that, include: The pattern extraction module is configured to scan the input satellite signal data, identify and extract all run-length encoded segments that meet the minimum length threshold, and divide the original data into alternating run-length encoded segments and non-run-length encoded gap sequences. The arithmetic sequence detection and encoding module is configured to perform arithmetic sequence detection on each of the non-run-length encoded gap sequences, and independently encode and output the arithmetic segments that meet the minimum length threshold in an arithmetic sequence pattern; The differential preprocessing module is configured to divide the remaining data that was not captured by the arithmetic sequence detection into blocks of fixed block size, and perform differential preprocessing on each data block. The optimal differential mode is selected from subtraction differential and XOR differential by a fast detection algorithm based on the highest byte zero rate. The multi-mode competitive encoding module is configured to synchronously perform rANS byte-plane encoding and differential bit packing encoding on each data block that has undergone differential preprocessing, and compare the number of output bytes with the original pass-through mode, and select the mode with the smallest encoding result as the optimal encoding scheme. The file output module is configured to write the global file header, block data area and block encoding sequence into the compressed file according to the multi-level compressed file format to form the final compressed output.

10. The system according to claim 9, characterized in that, The multi-mode competitive coding module includes: rANS byte plane encoder, configured to perform byte plane separation, per-plane tri-state decision and reverse rANS encoding; Differential bit packing encoder, configured to perform ZigZag encoding, bit width determination and fixed bit width packing; The original pass-through encoder is configured to perform direct writing of raw data and optimization of merging consecutive raw blocks. The contention selector is configured to compare the number of output bytes of the three encoders and select the optimal encoding result.

Images