Electromagnetic detection satellite data fast frame synchronization and analysis method based on parallel computing

Abstract

The application provides a kind of electromagnetic detection satellite data fast frame synchronization and analysis method based on parallelization calculation, belongs to remote sensing satellite data processing technical field, comprising: step 1: the original data stream of electromagnetic detection satellite is carried out data block and parallel frame synchronization processing, and the frame header position index table arranged according to the original order of sub-block is obtained;Step 2: based on the frame header position index table, the original data stream is carried out multi-load data parallel analysis processing, and the effective parameters of all frame data are extracted and recorded;Step 3: according to the effective parameters, the multi-load data after analysis is sorted and processed in parallel, and all the load data after processing is output through asynchronous parallel mode.Break through the technical bottleneck of traditional serial processing, realize the efficient parallel processing of mass data, improve the data processing speed and hardware resource utilization, while reducing the coupling of processing flow, meet the timeliness and expansibility requirements of satellite data processing.

Description

Technical Field

[0001] This invention relates to the field of remote sensing satellite data processing technology, and in particular to a method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing. Background Technology

[0002] As the first operational satellite in my country's geophysical field exploration satellite program, the electromagnetic detection satellite integrates nine payloads across three categories—electromagnetic field, in-situ ionospheric parameters, and ionospheric plasma structure—to meet the needs of geophysical field observation. A single mission generates massive amounts of data composed of multiple payloads, with downlink frames containing data from multiple payloads and exhibiting the inherent characteristic of individual data transmission frames containing only one payload stream. Frame synchronization and parsing of this type of multi-payload mixed data stream is a core pre-processing step in the application of electromagnetic detection satellite data, directly determining the timeliness and effectiveness of data processing.

[0003] Currently, the industry still uses traditional serial processing procedures for processing electromagnetic detection satellite data. This process exposes numerous technical bottlenecks when faced with massive mixed data streams from multiple payloads, making it unsuitable for the actual needs of satellite data processing: First, frame synchronization efficiency is extremely low. Conventional serial algorithms require scanning the data stream byte by byte to locate the frame header, which is time-consuming in massive data processing scenarios, becoming the primary efficiency bottleneck. Second, the coupling between multiple payloads is too high. Existing methods typically complete data parsing frame by frame sequentially, failing to achieve parallel separation and parsing of multiple payload data streams. The overall efficiency of data processing is limited by the processing speed of a single payload. Third, the deduplication and sorting of data within payloads is highly complex. Payload data needs to be ordered and redundant data removed based on frame counts, but traditional processing methods cannot effectively utilize CPU multi-core computing resources, resulting in low processing efficiency in this stage and further slowing down the overall data processing flow.

[0004] Existing research focuses primarily on optimizing the processing of single-payload satellite data, neglecting to address the core issue of efficient parallel processing of mixed data streams from multiple payloads in electromagnetic detection satellites. This results in significant technological gaps in current processing technologies, leading to a series of practical application shortcomings: First, the overall timeliness of data processing is insufficient. In scenarios with stringent requirements for data processing speed, such as disaster monitoring, the efficiency of traditional methods cannot meet the demands of real-time applications. Second, the utilization rate of hardware computing resources is low. The high-performance computing capabilities of multi-core CPUs are not fully utilized, resulting in a significant waste of computing resources. Finally, the overall processing flow is highly coupled, with each processing stage mutually constraining the others. If new payload data processing types are required, the entire process needs to be significantly adjusted, resulting in poor system scalability and compatibility.

[0005] In summary, to address the processing requirements of mixed data streams from multiple payloads in electromagnetic detection satellites, there is an urgent need to develop a novel frame synchronization and parsing method. Summary of the Invention

[0006] This invention provides a method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing, in order to solve the aforementioned technical problems.

[0007] This invention provides a method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing, comprising:

[0008] Step 1: Perform data block and parallel frame synchronization processing on the raw data stream of the electromagnetic detection satellite to obtain a frame header position index table arranged according to the original order of the sub-blocks;

[0009] Step 2: Perform parallel parsing of multi-payload data on the original data stream based on the frame header position index table, and extract and record the effective parameters of all frame data;

[0010] Step 3: Based on the valid parameters, sort and deduplicate the parsed multi-load data in parallel, and output all processed load data in an asynchronous parallel manner.

[0011] Preferably, step 1 includes:

[0012] The raw data stream from the electromagnetic detection satellite is logically divided into multiple sub-blocks according to a fixed block size;

[0013] Frame synchronization is based on OpenMP multi-threaded parallel processing of data within each sub-block. Each thread is responsible for several lines in a sub-block, while the SIMD instruction set is used to accelerate the matching and comparison of frame header identifiers.

[0014] Record the global offset of all frame headers and generate a frame header position index table arranged in the original order of the sub-blocks.

[0015] Preferably, atomic operations are used to update the frame header position index table between multiple threads, and the index table is maintained using a lock-free data structure.

[0016] Preferably, step 2 includes:

[0017] Two-stage parallel parsing is implemented using OpenMP multithreading;

[0018] Extract the data region of each frame based on the frame header offset in the frame header position index table, parse the payload type identifier and frame count identifier in the frame header, and record the valid parameters of all frame data.

[0019] The fast frame synchronization and parsing method for electromagnetic detection satellite data based on parallel computing according to claim 4 is characterized in that the two-stage parallel parsing includes:

[0020] The first level is sub-block level parallelism, where each thread is responsible for the overall parsing of several sub-blocks;

[0021] The second level is line-by-line parallelism within sub-blocks, where each thread is responsible for parsing several lines within a single sub-block.

[0022] Preferably, step 3 includes:

[0023] All parsed frame data are sorted as a whole according to the payload type identifier in the valid parameters, and the data is divided according to the payload type identifier.

[0024] Based on OpenMP, the payload data is processed in parallel using multiple threads. Each thread is responsible for several payloads and performs sorting and deduplication operations on the corresponding payload data according to the frame count identifier of each payload.

[0025] Asynchronous tasks are used to output all sorted and deduplicated payload data in parallel.

[0026] The present invention provides an electronic device, comprising: a processor and a memory, wherein the memory is used to store computer program code, the computer program code including computer instructions, and when the processor executes the computer instructions, the electronic device executes the aforementioned method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing.

[0027] The present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor of an electronic device, cause the processor to execute the aforementioned method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing.

[0028] Compared with the prior art, the beneficial effects of this application are as follows:

[0029] Breaking through the technical bottlenecks of traditional serial processing, this technology enables efficient parallel processing of massive amounts of data, improving data processing speed and hardware resource utilization, while reducing the coupling of the processing flow and meeting the timeliness and scalability requirements of satellite data processing.

[0030] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0031] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0032] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0033] Figure 1 This is a flowchart of a method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing, as described in an embodiment of the present invention.

[0034] Figure 2 This is a parallel architecture diagram in an embodiment of the present invention;

[0035] Figure 3 This is a performance comparison curve based on CPU utilization in an embodiment of the present invention. Detailed Implementation

[0036] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0037] This embodiment takes the processing of a 6.68GB mixed raw data stream from multiple payloads of an electromagnetic detection satellite as an example to provide a detailed description of the fast frame synchronization and parsing method for electromagnetic detection satellite data based on parallel computing described in this invention. The hardware computing platform of this embodiment is a 16-core CPU, adapted to the SSE / AVX series SIMD instruction set, and implemented with multi-threaded parallel programming based on OpenMP4.5. Those skilled in the art can adjust the relevant parameters according to the actual hardware configuration, and this invention does not limit this.

[0038] The method described in this invention constructs a three-level parallel architecture of data frame alignment, data parsing, and data organization, sequentially executing three core steps: data segmentation and parallel frame synchronization, parallel parsing of multi-payload data, and parallel sorting and deduplication of multi-payload data with asynchronous parallel output. Figure 1 As shown, the specific implementation methods for each step are as follows:

[0039] Step 1: Perform data block segmentation and parallel frame synchronization processing on the raw data stream of the electromagnetic detection satellite to obtain a frame header position index table arranged according to the original order of the sub-blocks:

[0040] This step is the data frame alignment process. Its core functionality involves segmenting the raw data stream into blocks and locating the frame headers in parallel, generating an ordered frame header position index table. This provides a precise basis for subsequent data parsing based on the frame header positions. Specifically, it includes the following sub-steps:

[0041] Step 1.1 Logical Data Stream Segmentation: The 6.68GB raw data stream from the electromagnetic detection satellite is logically segmented into fixed block sizes of 256MB / block. The segmentation process only divides the data stream by address, without physical copying, to avoid data redundancy. Ultimately, the raw data stream is segmented into 27 data sub-blocks (the first 26 sub-blocks are 256MB each, and the last sub-block is the remaining 12GB). If the hardware platform has better memory resources, the block size can be increased; if memory resources are limited, the block size can be decreased. The block size selection is based on matching the hardware cache and avoiding frequent memory swapping.

[0042] Step 1.2 Multi-threaded Parallel Frame Synchronization + SIMD Frame Header Matching Acceleration: Based on OpenMP multi-threading technology, 16 parallel threads are started to match the number of 16 CPU cores. Thread tasks are allocated using a sub-block allocation + row-level splitting method: First, 27 data sub-blocks are sequentially allocated to the 16 threads, with some threads handling two sub-blocks. Then, each data sub-block is split into fixed row sizes (1024 rows / segment in this embodiment), and each thread is responsible for frame synchronization processing of several rows within its allocated sub-block. During frame header localization, the SSE / AVX series SIMD instruction set is used to perform multi-byte parallel comparison of the frame header identifier. Traditional serial algorithms compare one byte at a time, while the SIMD instruction set can perform parallel comparison of 16 / 32 bytes simultaneously, significantly improving the matching speed of the frame header identifier. When a thread matches the frame header identifier, the local offset (offset within the sub-block) of the frame header is initially marked.

[0043] Step 1.3 Generate Frame Header Position Index Table: Each thread converts the marked local offset of the frame header into a global offset (global offset = sub-block start address + local offset), and submits the global offset of the frame header, the sub-block number to which the frame header belongs, and other information to the frame header position index table. This step uses atomic operations to update the index table between multiple threads, and maintains the index table through a lock-free data structure to avoid contention caused by multiple threads updating the index table simultaneously, thus ensuring the accuracy of the index table data. After all threads complete the frame synchronization processing, they sort the frame header position index table according to the original order of the sub-blocks to generate an ordered frame header position index table containing all global offsets of the frame header, completing the data frame alignment process.

[0044] Step 2: Based on the frame header position index table, perform parallel parsing of multi-payload data on the original data stream, extract and record the valid parameters of all frame data:

[0045] This step is the data parsing stage. Based on the frame header position index table generated in step 1, it implements two-level parallel parsing of multi-payload data, extracts and records the effective parameters of the frame data, and provides a foundation for subsequent data processing. Specifically, it includes the following sub-steps: Step 2.1: OpenMP Two-Level Parallel Parsing: Based on OpenMP multi-threading technology, 16 parallel threads are started to implement two-level parsing: sub-block level parallelism and row level parallelism. The first level is sub-block level parallelism, where the 27 data sub-blocks cut in step 1 are allocated to the 16 threads, with some threads responsible for the overall parsing of two sub-blocks. The second level is row level parallelism, where each data sub-block is split into 1024 rows / segments, and each thread is responsible for the fine-grained parsing of several rows within its assigned sub-block. The task allocation method for the two-level parallel parsing is consistent with the frame synchronization task allocation in step 1, ensuring the balance of thread tasks and avoiding overall efficiency degradation caused by single-thread overload.

[0046] Step 2.2: Frame Data Extraction and Valid Parameter Parsing: Each thread accurately extracts the data region of each frame within the corresponding data sub-block based on the global offset of the frame header in the frame header position index table, avoiding invalid data scanning across sub-blocks. The frame header of each frame is parsed to extract core valid parameters such as the payload type identifier and frame count identifier, while also recording auxiliary parameters such as the frame data length and data start address. Each thread stores the parsed valid parameters of the frame data in frame header order into a unified parameter database. The parameter database also uses a lock-free data structure, and atomic operations are used to achieve parallel writing by multiple threads, ensuring the accuracy and orderliness of parameter recording. In this step, since a single data transmission format frame of the electromagnetic detection satellite contains only one payload data stream, the payload type identifier can directly associate a single frame of data with its corresponding payload, providing a precise basis for subsequent separation and processing of multiple payload data.

[0047] Step 3: Based on the valid parameters, sort and deduplicate the parsed multi-load data in parallel, and output all processed load data in an asynchronous parallel manner:

[0048] This step is the data preparation stage, which separates multi-load data, performs parallel sorting and deduplication, and outputs asynchronously, completing the entire data processing flow. It specifically includes the following sub-steps:

[0049] Step 3.1: Payload data sorting and partitioning: Based on the payload type identifier obtained in Step 2, all frame data in the parameter database are globally sorted, and the frame data are classified and partitioned according to the payload type. The mixed data stream is separated into 9 independent payload data streams (corresponding to the 9 effective payloads of the electromagnetic detection satellite). Each payload data stream contains all frame data of that payload and corresponding frame count identifiers, data addresses and other parameters, thereby achieving decoupling of multiple payload data.

[0050] Step 3.2 Multi-threaded Parallel Sorting and Deduplication: Based on OpenMP multi-threading technology, 16 parallel threads are started, and the 9 payload data streams are distributed to the 16 threads (some threads are responsible for multiple payloads, or reserved threads are reserved to deal with sudden data bursts). Each thread is responsible for sorting and deduplicating the payload data stream it is assigned to. Each thread sorts the frame data of its payload data stream in ascending order based on the frame count identifier, so that the payload data is arranged in the order of the frame count. At the same time, the redundant data corresponding to duplicate frame counts is removed by comparing the frame count identifiers, and only the unique frame data is retained, thus completing the processing of the payload data. In this embodiment, the payload data deduplication accuracy achieved by this step is ≥99.9%, which effectively ensures the integrity of the data.

[0051] Step 3.3 Asynchronous Task Parallel Output: For the 9 payload data channels that have been sorted and deduplicated, parallel output is achieved using asynchronous tasks. An independent asynchronous output task is assigned to each payload data channel. The output task is independent of the data processing thread, avoiding the output contention problem caused by multiple threads writing data to the output port at the same time. During the output process, the data is identified according to the payload type, and the payload data, along with the corresponding frame count identifier and parameter information, are synchronously output to the specified storage medium / application interface. The output format is adapted to the application requirements of subsequent satellite data, and the output format can be adjusted according to actual needs. This invention does not limit this.

[0052] This embodiment uses the above method to process the raw data stream of a 6.68GB electromagnetic detection satellite. The entire process runs on a 16-core CPU platform, and the processing results verify the effectiveness of the invention. Figure 3 As shown:

[0053] The overall processing time was 33.665874 seconds, which is about 6.03 times faster than the traditional serial processing method of 203 seconds, achieving a speedup of 5-7 times.

[0054] The CPU core utilization rate of the frame header synchronization stage and the data parsing stage is stable at over 99%, which fully taps the CPU's multi-core computing power and significantly improves the utilization rate of hardware resources.

[0055] The deduplication accuracy of all 9 load data streams is ≥99.9%, and the data sorting order is 100%, ensuring the integrity and accuracy of the data.

[0056] Multi-load data processing is fully decoupled. Adding or removing load data processing types only requires adjusting the task logic of the corresponding thread, without modifying the overall parallel architecture, which greatly improves system scalability.

[0057] The entire process is free from multi-threaded contention and output contention issues, ensuring stable processing and strong robustness.

[0058] like Figure 2 The diagram shown is a parallel architecture diagram of the fast frame synchronization and parsing method for electromagnetic detection satellite data based on parallel computing provided by the present invention.

[0059] In this embodiment, the electromagnetic detection satellite data transmission frame adopts a fixed format frame structure, and the core identifier is defined as follows:

[0060] Frame header identifier: fixed as a 4-byte hexadecimal number 0xEB90, located at the offset of 0–3 bytes from the beginning of each frame data, used for frame header positioning;

[0061] Payload type identifier: 1-byte unsigned integer, located in the 4th byte after the frame header, with a value range of 1–9, corresponding to the 9 types of satellite payloads, used to distinguish the payloads belonging to a single frame;

[0062] Frame count identifier: A 4-byte unsigned integer located in the 5th–8th bytes after the frame header. It is a unique timing number for the payload data frame and is used for frame data sorting and deduplication.

[0063] Fixed frame length: 1024 bytes. Each frame contains only one payload data channel and there is no case of multiple payloads being mixed.

[0064] In this embodiment, the parallel comparison of the SIMD instruction set is specifically implemented as follows:

[0065] Frame header matching uses the AVX2 instruction set to achieve 128-bit / 256-bit multi-byte parallel comparison. The specific execution logic is as follows:

[0066] Load 32 consecutive bytes of data within the sub-block into AVX2 register ymm0, and expand the frame header identifier 0xEB90 into a 32-byte matching template and load it into ymm1;

[0067] The parallel comparison instruction _mm256_cmpeq_epi8 is executed to complete the full byte matching of 32 bytes with the frame header template in one go;

[0068] If the comparison result contains an all-1 mask, it is determined that the frame header has been hit, and the current byte offset is recorded; if it has not been hit, the scanning continues with a sliding window of 32 bytes in step size, which is 32 times more efficient than single-byte serial comparison.

[0069] In this embodiment, the specific implementation of atomic operations and lock-free data structures is as follows:

[0070] The frame header position index table is stored using a lock-free dynamic array, and multi-threaded updates use CAS atomic operations.

[0071] Index table storage structure: [sub-block number, global offset, frame length, valid flag];

[0072] When a thread writes to the index table, it first uses a CAS atomic operation to preempt the write position. If successful, the data is written; if it fails, it is retried. There is no mutex lock throughout the process, which avoids thread blocking.

[0073] The index table is sorted in ascending order by sub-block number to ensure that the frame header position order is consistent with the original data stream.

[0074] In this embodiment, OpenMP's two-level parallel parsing employs a load balancing scheduling strategy, with the following specific rules:

[0075] Sub-block parallelism: The 27 sub-blocks are automatically allocated according to thread load using #pragmaompparallelforschedule(dynamic) to avoid thread idleness;

[0076] Row-level parallelism: Each sub-block is divided into 1024 rows, and static scheduling is used to ensure that the processing time of each row of data is equal, thus ensuring thread load balance.

[0077] Thread count configuration: omp_set_num_threads(16), consistent with the number of physical CPU cores, disables hyper-threading interference, and ensures computational efficiency.

[0078] In this embodiment, frame parameter parsing is strictly performed based on a fixed intra-frame offset:

[0079] Load type identifier: offset 4 bytes, read 1 byte and map to load number 1–9;

[0080] Frame count identifier: offset 5–8 bytes, parsed as a 32-bit unsigned integer in big-endian order;

[0081] The parameter storage uses a lock-free hash table, with the frame count identifier as the key and the payload type, data address, and frame length as the value. Multi-threaded writes ensure uniqueness through atomic operations, preventing data conflicts.

[0082] In this embodiment, each payload data uses a combination of frame count identifier and hash deduplication scheme:

[0083] Sorting: OpenMP parallel fast sorting is used, which sorts the frames in ascending order by frame count identifier. The frames are then sorted in segments using #pragmaompparallelfor and then merged.

[0084] Deduplication: Construct a frame count → frame address hash mapping. When traversing frame data, if a frame count already exists, mark it as a duplicate frame and remove it. Deduplication accuracy is ≥99.9%.

[0085] Each load is processed independently, with no data dependency between loads, achieving complete decoupling.

[0086] In this embodiment, asynchronous output uses a thread pool and an independent output queue to achieve contention-free output:

[0087] Independent output queues and asynchronous tasks are created for the nine payloads. The tasks are executed by an independent thread pool, separate from the data processing thread.

[0088] Output rules: Write in ascending order by frame count, single-load output is serial, multi-load output is parallel, and there are no cross-load write conflicts;

[0089] The output interface supports file storage / network push, and the output format is a payload-specific binary format with frame count check bits.

[0090] In this embodiment, the method includes complete fault-tolerant logic to ensure processing stability:

[0091] Frame header mismatch: When the fixed frame header identifier 0xEB90 is not detected for 3 consecutive data sub-blocks, it is automatically determined that the data is corrupted, the current sub-block is skipped and written to the error log (the log contains the sub-block number, starting address and exception type), and the next sub-block is processed.

[0092] Thread abnormal exit: When an OpenMP child thread terminates abnormally, the main process catches the SIGSEGV / SIGABRT signal, immediately releases the resources occupied by the thread, and reallocates the unfinished tasks to idle threads without interrupting the overall processing flow;

[0093] Data out of bounds anomaly: Verification is performed based on a fixed frame length of 1024 bytes. If the length of the extracted frame data exceeds 1024 bytes, it is automatically truncated to 1024 bytes and marked as an invalid frame, which will not participate in subsequent parsing and sorting.

[0094] Output error: If the output queue is blocked for more than 1 second, the data to be output will be automatically cached in a temporary memory buffer. After the output port is restored, the data will be transmitted in the order of frame count to ensure no data loss.

[0095] Those skilled in the art should understand that the method described in this invention is not only applicable to a data size of 6.68GB and a 16-core CPU platform, but can also adjust parameters such as data block size, number of OpenMP threads, SIMD instruction set type, and row-level split size according to the actual electromagnetic detection satellite data size and hardware computing platform (such as 8-core or 32-core CPU). All parameter adjustments and local optimizations based on the core technical solution of this invention are within the protection scope of this invention.

[0096] The present invention provides an electronic device, comprising: a processor and a memory, wherein the memory is used to store computer program code, the computer program code including computer instructions, and when the processor executes the computer instructions, the electronic device executes the aforementioned method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing.

[0097] The present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor of an electronic device, cause the processor to execute the aforementioned method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing.

[0098] This invention provides a method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing, and further includes:

[0099] The collaborative optimization of data segmentation and parallel frame synchronization includes adaptive segmentation control driven by frame header distribution information entropy, dynamic balancing scheduling of thread load entropy weights, and incremental hash merging of lock-free index tables. Specifically, it includes:

[0100] Step 01: Adaptive block control driven by frame header distribution information entropy:

[0101] Real-time acquisition of 3D feature parameters: CPU core utilization U (dimensionless, ), Sub-block payload data density (Unit: frames / MB), frame header spatial distribution frequency f (unit: times / KB), calculate the frame header distribution information entropy value H to quantify the uniformity of sub-block data, the information entropy calculation formula is:

[0102] ,in, This represents the percentage of frame headers within the i-th frame header distribution interval. Let i be the frame header number of the i-th interval. is the total number of frame headers in the sub-block; n is the number of distribution intervals, ranging from 8 to 16.

[0103] It needs to be explained that when When = 0, it is agreed that =0.

[0104] Based on the coupled mapping relationship between information entropy and three-dimensional feature parameters, the optimal logical block size is dynamically determined. :

[0105] ,in, The base block size (unit: MB, value is 256MB); The maximum information entropy (dimensionless) when the frame header is uniformly distributed. , , The coupling weight coefficients (dimensionless) =0.8, =0.6, =0.7); Global average load data density (unit: frames / MB). The global average frame header distribution frequency (unit: times / KB);

[0106] Constraints: This ensures that the block size is adapted to the hardware cache and memory bandwidth.

[0107] Step 02: Dynamic load balancing scheduling of threads:

[0108] An entropy weight method is used to construct a thread load balancing scheduling model, calculating the thread load entropy weight and task allocation weight to achieve unequal distribution of sub-block and row-level processing tasks.

[0109] Calculate the normalized value of thread load : ,in, The current load rate of the i1th thread (dimensionless). ), , These represent the maximum and minimum thread loads, respectively.

[0110] It needs to be explained that when When all threads have completely identical load, the following convention applies. =0.5, to avoid the risk of division by zero.

[0111] Constructing the load probability matrix ,in, (m is the total number of threads);

[0112] Calculate the thread load information entropy E and entropy weight : , ,in, Let i1 be the load entropy value of the i1th thread. The entropy weight of the i1th thread (dimensionless) );

[0113] when When = 0, it is agreed that =0.

[0114] Determine task allocation weights Combined with OpenMP dynamic scheduling strategy to allocate sub-blocks and row-level tasks: ,in, Delay weighting coefficients for frame header processing (dimensionless, The delay is determined by the ratio of the current frame header processing delay to the average delay; constraints: This ensures that no task is missed in the allocation of tasks and eliminates thread load imbalance and idle computing resources.

[0115] In this embodiment, The deviation between the efficiency of single-threaded frame header processing and the global average efficiency is used to quantify the following calculation steps:

[0116] Real-time acquisition of the actual processing latency for each thread to complete sub-block frame synchronization (Unit: ms);

[0117] Calculate the average processing latency of all threads (Unit: ms) ;

[0118] Calculate the delay ratio ;

[0119] Determined according to the following rules :

[0120] like (Thread processing speed is faster than average). =1.5;

[0121] If 0.8≤ ≤1.2 (Thread processing speed is close to average). =1.0;

[0122] like >1.2 (Thread processing speed is slower than average). =0.5.

[0123] Step 03: Incremental hash merging of lock-free index tables:

[0124] A lock-free hash index structure is constructed based on CAS atomic operations. Segmented incremental writing of frame header position information is performed, and conflict-free merging of multi-threaded index fragments is completed through a globally ordered concatenation algorithm of the index table.

[0125] Hash index key-value calculation: ,in, This is a byte concatenation operation, where Key is the hash index key (dimensionless, 32-bit integer).

[0126] CAS atomic write determination: ;

[0127] If the CAS operation succeeds, write the frame header information structure FrameInfo (containing sub-block number, global offset, and frame length); if it fails, perform hash chain collision resolution and append the conflicting frame header information to the collision chain of the corresponding hash bucket.

[0128] Globally ordered concatenation of index table with incremental length : ,in, This represents the total number of frame headers detected by a single thread. The number of frame header entries written in a single incremental operation (rounded down).

[0129] When the calculation result At that time, take Ensure that at least one frame header information is written at a time; when At that time, take To avoid write overflow, the more single-threaded frame headers, the closer the CPU utilization is to the optimal load of 99%, and the more uniform the frame header distribution, the larger the single incremental write length can be. This allows for reduced overhead from thread scheduling and global sorting through batch writing. Conversely, reducing the write length ensures write stability. Boundary constraints prevent write overflow or meaningless zero-value writes, forming a complete logical closed loop with lock-free hash writing and globally ordered concatenation of the index table.

[0130] After the splicing is completed, the full index table is sorted in ascending order by the sub-block number to generate a globally ordered frame header position index table, realizing lock-free, adaptive, and high-throughput parallel processing of the entire frame synchronization process.

[0131] In this embodiment, when the amount of data to be processed is ≥1GB, the frame header distribution information entropy adaptive block division is automatically enabled; when the amount of data is <1GB, a fixed 256MB block is used to balance efficiency and memory usage.

[0132] When CPU core utilization is <95%, thread load entropy weight dynamic balancing scheduling is automatically enabled; when utilization is ≥95%, fixed task allocation is maintained to reduce scheduling overhead.

[0133] When the frame header matching error rate is greater than 0.1%, multi-level CRC recursive pre-checking of the frame header is automatically enabled; when the error rate is less than or equal to 0.1%, SIMD frame header matching is performed directly to reduce computational overhead.

[0134] The beneficial effects of the above technical solution are as follows: by using information entropy-driven adaptive block partitioning, entropy weight dynamic scheduling, and lock-free hash incremental merging, the entire process of frame synchronization is made adaptive and lock-free, further controlling the CPU core utilization fluctuation within 0.3%, reducing frame synchronization latency by 25% to 30% compared to the basic solution, increasing parallel throughput by 35%, completely eliminating thread load skew and index table update blocking problems, and significantly improving the stability and efficiency of massive multi-load data frame synchronization.

[0135] This invention provides a method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing, and further includes: constructing a multi-level CRC recursive pre-check of the frame header, optimizing the alignment of the CPU multi-level cache hierarchy, and performing end-to-end optimization of frame counting timing smoothing flow control and payload out-of-order compensation output, specifically including:

[0136] Step 001: Multi-level CRC recursive pre-check in frame header:

[0137] Before multi-byte parallel frame header matching in the SIMD instruction set, a two-level recursive pre-check of the frame header using CRC16 / CRC32 is performed, and valid data segments are dynamically and adaptively filtered based on the check threshold.

[0138] CRC recursive check calculation:

[0139] ,in, This is the CRC16 check result after the s-th byte of data has been processed. This is the CRC16 check result after the (s-1)th byte has been processed; This is the logical right shift operator; This is the bitwise XOR operator; For the pre-computed CRC16 lookup table; This is the bitwise AND operator; It is an 8-bit all-1 mask; The s-th byte of the data stream to be verified;

[0140] It should be noted that the initial value of CRC16 =0xFFFF, using the standard CRC16-CCITT algorithm. This is a pre-computed 256-byte standard lookup table.

[0141] CRC32 recursion formula:

[0142] ,in, This is the CRC32 check result after the s-th byte of data has been processed. This is the CRC32 check result after the (s-1)th byte has been processed; For the pre-computed CRC32 lookup table;

[0143] It should be noted that the initial value of CRC32 =0xFFFFFFFF, using the standard IEEE 802.3 CRC32 algorithm. This is a pre-computed 256-byte standard lookup table.

[0144] Dynamic adaptive verification threshold : ,in, The baseline verification threshold (value 0xFFFF, corresponding to the CRC16 verification pass standard, dimensionless). , , These are threshold correction coefficients, with values ​​of 0.02, 0.01, and 0.005 respectively.

[0145] It should be noted that, ≤0xFFFF, 0xFFFF is used when exceeding the upper limit; CRC32 check threshold is 16, matching a 32-bit width.

[0146] Only when CRC16 > And CRC32 > If a data segment is deemed valid, it is entered into a SIMD multi-byte parallel comparison; otherwise, noisy data and invalid segments are discarded to reduce the computational overhead of SIMD.

[0147] Step 002: CPU multi-level cache level alignment optimization:

[0148] Based on the physical capacity and cache line granularity of the CPU's L1 / L2 / L3 caches, tiered cache alignment is implemented to minimize cache misses and memory access latency. The core formula is as follows:

[0149] Tiered cache alignment length : ,in, This is the size of the L-th level cache line (64B when L=1, 64B when L=2, and 64B when L=3). The length of a single frame of data (unit: B, value is 1024B). To round up; perform hierarchical alignment on the three types of core data:

[0150] Sub-block raw data: aligned to L2 cache line, ;

[0151] Frame header position index table: aligned to L1 buffer line. (Single index 32B);

[0152] Payload resolution parameters: aligned to L3 cache line. (Single parameter 64B);

[0153] Cache miss rate Optimization constraints: ,in, The maximum allowable miss rate for the L-th level cache (L1=5%, L2=10%, L3=15%). The total capacity of the Lth level cache (unit: B); by aligning storage, the miss rate of each level cache meets the constraints, and adapts to the continuous read and write rules of SIMD registers.

[0154] Step 003: Frame counting timing smoothing flow control and load out-of-order compensation:

[0155] A timing smoothing model is constructed based on the frame count continuity coefficient. Dynamic bandwidth allocation and flow control are performed on the asynchronous output channel. Combined with the load out-of-order compensation algorithm, frame group batch aggregation and timing correction output are completed.

[0156] Frame count continuity coefficient : ,in, , These are the maximum and minimum values ​​of the j2-th load frame count, respectively. The total number of frames for the j2th channel payload (dimensionless, 0 ≤ ... ≤1);

[0157] It should be noted that when When =1 (single frame data), it is agreed that ,when When (invalid data) is encountered, the following convention applies. 0, constraint When frame counts are completely continuous = The molecule is 0. =1 represents optimal temporal continuity; the more severe the frame count loss and out-of-order issues, the better. The closer the coefficient is to 0, the worse the timing continuity. This coefficient accurately quantifies the timing characteristics of the load data, providing a core basis for subsequent bandwidth allocation and out-of-order compensation, and forming a complete logical connection with the asynchronous parallel output stage.

[0158] Dynamic bandwidth allocation for asynchronous output channels : ,in, Total output bandwidth (unit: MB / s). Total number of frames; via Dynamically adjust the output bandwidth of each load to ensure smooth timing;

[0159] It should be noted that, .

[0160] Payload out-of-order compensation frame group size : , among which, among which, The base frame group size (value is 256 frames). The global average frame count continuity coefficient; when When the values ​​are below average, increase the frame group size to compensate for out-of-order delivery; when When the value is close to 1, reduce the frame group size to ensure real-time performance; constraint: 32 ≤ ≤512, completes batch aggregation of frame groups and timing correction output, completely eliminating IO contention and timing disorder problems of multi-load parallel output.

[0161] In this embodiment, based on actual testing on a 16-core CPU platform, the cache alignment optimization of this invention shows that:

[0162] L1 cache miss rate ≤3.2% (meets the ≤5% constraint);

[0163] L2 cache miss rate ≤7.8% (meets the ≤10% constraint);

[0164] The L3 cache miss rate is ≤11.5% (meets the ≤15% constraint); it fully meets the cache miss rate constraint, reduces memory access latency by more than 40%, and improves the sequential read and write efficiency of SIMD instructions by 35%.

[0165] The beneficial effects of the above technical solution are as follows: by constructing a full-link closed-loop optimization system through multi-level CRC recursive pre-check, three-level cache alignment, frame counting timing smoothing flow control and load out-of-order compensation, the amount of invalid SIMD computation is reduced by 75%, the cache miss rate is reduced by 80%, and the IO competition and timing disorder problems of multi-load parallel output are completely eliminated. On the basis of the 5-7 times speedup of the basic solution, the overall processing speed is further improved by 25%~30%, which greatly improves the robustness and processing efficiency of the system.

[0166] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing, characterized in that, include: Step 1: Perform data block and parallel frame synchronization processing on the raw data stream of the electromagnetic detection satellite to obtain a frame header position index table arranged according to the original order of the sub-blocks; Step 2: Perform parallel parsing of multi-payload data on the original data stream based on the frame header position index table, and extract and record the effective parameters of all frame data; Step 3: Based on the valid parameters, sort and deduplicate the parsed multi-load data in parallel, and output all processed load data in an asynchronous parallel manner.

2. The method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing according to claim 1, characterized in that, Step 1 includes: The raw data stream from the electromagnetic detection satellite is logically divided into multiple sub-blocks according to a fixed block size; Frame synchronization is based on OpenMP multi-threaded parallel processing of data within each sub-block. Each thread is responsible for several lines in a sub-block, while the SIMD instruction set is used to accelerate the matching and comparison of frame header identifiers. Record the global offset of all frame headers and generate a frame header position index table arranged in the original order of the sub-blocks.

3. The method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing according to claim 2, characterized in that, Atomic operations are used to update the frame header position index table between multiple threads, and the index table is maintained using a lock-free data structure.

4. The method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing according to claim 1, characterized in that, Step 2 includes: Two-stage parallel parsing is implemented using OpenMP multithreading; Extract the data region of each frame based on the frame header offset in the frame header position index table, parse the payload type identifier and frame count identifier in the frame header, and record the valid parameters of all frame data.

5. The method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing according to claim 4, characterized in that, The two-level parallel parsing includes: The first level is sub-block level parallelism, where each thread is responsible for the overall parsing of several sub-blocks; The second level is line-by-line parallelism within sub-blocks, where each thread is responsible for parsing several lines within a single sub-block.

6. The method for fast frame synchronization and parsing of electromagnetic detection satellite data based on parallel computing according to claim 4, characterized in that, Step 3 includes: All parsed frame data are sorted as a whole according to the payload type identifier in the valid parameters, and the data is divided according to the payload type identifier. Based on OpenMP, the payload data is processed in parallel using multiple threads. Each thread is responsible for several payloads and performs sorting and deduplication operations on the corresponding payload data according to the frame count identifier of each payload. Asynchronous tasks are used to output all sorted and deduplicated payload data in parallel.

7. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein when the processor executes the computer instructions, the electronic device performs the fast frame synchronization and parsing method for electromagnetic detection satellite data based on parallel computing as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which includes program instructions that, when executed by a processor of an electronic device, cause the processor to perform the fast frame synchronization and parsing method for electromagnetic detection satellite data based on parallel computing as described in any one of claims 1 to 6.

Images