Real-time spectral data stream processing method and system based on multi-process pipeline

Abstract

The present application belongs to the technical field of digital signal processing, and in particular to a real-time spectrum data stream processing method and system based on a multi-process pipeline, comprising the following steps: S1, a three-stage pipeline processing architecture is built, which is composed of a data acquisition stage, an FFT calculation stage and a spectrum post-processing stage; the FFT calculation stage is configured with a plurality of working processes, which separately complete FFT transformation and spectrum analysis operations; S2, a plurality of spectrum data is packaged to form batch data packets, and unified transmission between processes is completed in units of batch data packets; S3, each working process is configured with an independent task queue, and an idle working process grabs a to-be-processed task from the tail of the task queue of the remaining working processes; S4, the serial processing efficiency of the system is monitored in real time, and the number of active working processes in the FFT calculation stage is dynamically adjusted according to the efficiency threshold value. The present application can solve the technical problems of the existing embedded platform real-time spectrum analysis, such as Python GIL limitation leading to real parallelism being impossible, low multi-core resource utilization, and large process communication overhead.

Description

Technical Field

[0001] This invention belongs to the field of digital signal processing technology, and in particular to a real-time spectrum data stream processing method and system based on a multi-process pipeline. Background Technology

[0002] Spectrum analysis is a key supporting technology in fields such as software-defined radio, radio astronomy, wireless communication monitoring, and electromagnetic compatibility testing. With the evolution of wireless communication technology, the instantaneous bandwidth of the signals to be processed is constantly increasing, placing higher demands on the real-time processing capabilities, computational efficiency, and deployment flexibility of spectrum analysis systems. In particular, implementing broadband real-time spectrum analysis on low-cost, low-power embedded platforms such as the Raspberry Pi faces the following prominent technical challenges: (1) Conflict between computational load and real-time performance The Fast Fourier Transform (FFT), as the core operation of spectrum analysis, has a computational complexity of O(NlogN). When processing wideband signals, large-point FFT operations place a huge burden on the processor's computing power. Traditional single-threaded serial processing architectures, where data acquisition, FFT calculation, and post-processing are executed sequentially, will have any delay or blockage in any stage accumulate in the entire processing pipeline, leading to increased overall system latency and decreased real-time throughput, making it difficult to meet the real-time processing requirements of wideband signals.

[0003] (2) Low resource utilization of multi-core processors In embedded hardware environments such as Raspberry Pi, which commonly feature multi-core CPUs, existing technologies generally fail to fully exploit the potential of parallel computing. The modular scheduling scheme based on GNU Radio, although it supports multi-threading, is prone to buffer overflows due to processing delays when dealing with sudden data streams, making it difficult to guarantee data integrity.

[0004] The traditional serial solution based on RTL-SDR and DSP server adopts a single-threaded serial processing mode, and the actual overall CPU utilization is usually less than 40%, resulting in a serious waste of multi-core computing resources.

[0005] CUDA-based GPU parallel acceleration solutions: Although FFT operation speed can be significantly improved through GPU, the solution is limited by the data transfer bandwidth and latency overhead between the CPU and GPU. This solution not only consumes a lot of power, but is also not suitable for embedded micro platforms that lack high-performance discrete GPUs.

[0006] (3) Parallelism limitations of the Python runtime environment In embedded Python development, developers commonly face the constraint of the Global Interpreter Lock (GIL). The existence of the GIL prevents programs developed based on traditional multithreading modules from achieving true parallel execution across multiple physical cores when handling CPU-intensive tasks such as FFT. Experimental data shows that the actual performance improvement of multithreading solutions in CPU-intensive scenarios is typically less than 1.2 times.

[0007] (4) Bias towards multithreading and multiprocessing technologies: The Global Interpreter Lock (GIL) is an inherent constraint of the Python runtime environment. In multithreaded mode, only one thread can execute Python bytecode at a time, and multithreaded programs cannot utilize multi-core resources when handling CPU-intensive tasks such as FFT.

[0008] In the Python development community, multi-processing has long been considered unsuitable for high-frequency data stream processing. This perception stems from the performance limitations of the `multiprocessing` module prior to Python 3.2—the pickle serialization mechanism was inefficient, with measured latency for single data transfers between processes reaching milliseconds. Although subsequent Python versions have continuously optimized the `multiprocessing` module (such as introducing more efficient serialization protocols and shared memory mechanisms), the perception that "multi-processing is unsuitable for real-time processing" remains prevalent in the embedded Python development community. This perception is reflected in discussions on technical blogs, open-source project documentation, and developer forums: when choosing a real-time data stream processing framework, developers still prioritize multi-threaded solutions (despite being limited by the GIL) or directly switch to C / C++, rarely attempting to leverage the advantages of multi-processing by optimizing IPC mechanisms.

[0009] As a result, Python developers face a dilemma in embedded real-time scenarios: accept the performance limitations imposed by the GIL, or abandon the Python ecosystem and switch to compiled languages.

[0010] (5) Power consumption and thermal constraints of embedded platforms Embedded devices such as Raspberry Pi typically have strict thermal power consumption (TDP) limits (approximately 5-8W). Under high workloads, the processor temperature rises rapidly, triggering hardware throttling protection mechanisms, which in turn leads to an irreversible decrease in processing performance. Existing technologies lack the ability to dynamically sense and manage processor temperature and load, making it difficult to achieve an optimal balance between performance and power consumption.

[0011] In summary, existing technologies have failed to effectively address the issues of parallel computing efficiency, pipeline congestion, and multi-core resource utilization in real-time spectrum analysis on embedded platforms. Summary of the Invention

[0012] The technical problem to be solved by this invention is to provide a real-time spectrum data stream processing method and system based on a multi-process pipeline, which aims to solve the technical problems in the existing embedded platform real-time spectrum analysis, such as the inability to achieve true parallelism due to Python GIL limitations, low utilization of multi-core resources, and high inter-process communication overhead.

[0013] To address the above problems, the technical solution adopted in this invention is: a real-time spectrum data stream processing method based on a multi-process pipeline, comprising the following steps: S1. Construct a three-stage pipelined processing architecture consisting of a data acquisition stage, an FFT computation stage, and a spectrum post-processing stage; the FFT computation stage is configured with several worker processes, each worker process is configured with an independent Python interpreter and an independent memory space, and performs FFT transformation and spectrum analysis operations independently; data interaction between the stages of the pipeline is completed through process-safe queues; S2. Pack multiple frames of spectrum data into batch data packets, and complete the unified transmission between processes in units of batch data packets; S3. Each worker process is configured with an independent task queue. Idle worker processes grab tasks to be processed from the tail of the task queues of other worker processes to evenly distribute the computing load of each process. S4. Monitor the serial processing efficiency of the system in real time and dynamically adjust the number of active working processes within the FFT computing level based on the efficiency threshold. When the serial processing efficiency is higher than the first threshold, it is determined to be a light load and the number of active working processes is reduced. When the serial processing efficiency is lower than the second threshold, it is determined to be a heavy load and the number of active working processes is increased to the preset limit.

[0014] Furthermore, in step S1, the total number of working processes is taken as the number of physical CPU cores minus 1. The processor is a quad-core ARM processor, which is configured with 3 working processes.

[0015] Furthermore, in step S2, the number of data frames encapsulated in the batch data packets is adaptively adjusted according to the data source output rate: when the data source output rate is higher than 200 frames / second, 30 to 50 frames are encapsulated in a single batch; when the data source output rate is lower than 100 frames / second, 10 frames are encapsulated in a single batch.

[0016] Furthermore, in step S2, a single batch of data packets completes two inter-process communications: one transmission and one result return. When 20 frames are used as the single batch encapsulation standard, the total number of inter-process communications for N frames of data is 0.1N.

[0017] Furthermore, in step S3, each working process is configured with a dual-end task queue. The local working process reads tasks from the head of the queue, and the idle process that steals tasks reads tasks from the tail of the queue. A random polling mechanism is used to select the queue of tasks to be stolen. The random polling mechanism is as follows: after removing the currently idle process, the numbers of all remaining working processes are randomly shuffled, and tasks are attempted to be stolen in the shuffled order. The polling stops when a valid task is obtained, until the entire task queue has been traversed.

[0018] Further, in step S4, the formula for calculating the serial processing efficiency is: Serial processing efficiency = (Serial processing output frame rate ÷ Data source output frame rate) × 100%; the first threshold is set to 60%, and the second threshold is set to 30%.

[0019] Furthermore, the system serial processing efficiency monitoring cycle is 1000 milliseconds. When the threshold condition is met for two consecutive monitoring cycles, the number of working processes is adjusted.

[0020] A real-time spectrum data stream processing system based on a multi-process pipeline includes: The data acquisition module connects to an external spectrum data source at its input end to read IQ sampling data or raw spectrum data and outputs a single-frame data stream. The batch packaging module connects to the output of the data acquisition module at its input end. It has a built-in data buffer that caches single-frame data streams. Once the number of data frames in the buffer reaches a preset number, it is packaged into batch data packets and output. The task queue module uses a multi-process safe queue to cache batch data packets, thereby decoupling the data production and computation processes. The computation module contains several worker processes, each configured with an independent Python interpreter and independent memory space, and independently performs FFT transformation and spectrum analysis operations. The job stealing scheduler establishes a control connection with all worker processes, monitors the task queue storage of each process in real time, and when it detects that the worker process queue is empty, it grabs tasks from the tail of other process queues and assigns them to the idle process. The results collection module connects to the output of all working processes, summarizes the spectrum processing results, statistically analyzes system performance indicators, and generates operation logs. The load-aware scheduler collects data on the data source output rate and serial processing frame rate, outputs control signals to the computing module, and dynamically adjusts the number of active working processes within the computing module based on the serial processing efficiency.

[0021] Furthermore, the computing module is divided into three computing modes: the light mode only performs signal peak detection; the medium mode performs FFT transformation operation; and the heavy mode simultaneously completes FFT transformation, multi-peak identification, noise floor solution and spectral feature extraction. The load-aware scheduler switches the corresponding working mode according to the real-time calculated load density.

[0022] Furthermore, the job-stealing scheduler is configured with dual-end task queue scheduling logic, where the local process reads tasks from the head of the queue, the idle process steals tasks from the tail of the queue, and a random polling mechanism is used to select the target queue for stealing. Furthermore, the load-aware scheduler adjusts the number of active worker processes using the following rules: serial processing efficiency = (serial processing output frame rate ÷ data source output frame rate) × 100%, with a first threshold of 60%, a second threshold of 30%, and a monitoring period of 1000 milliseconds. Adjustment is performed when the threshold conditions are met for two consecutive monitoring periods.

[0023] The beneficial effects of this invention are: 1. Multi-process architecture bypasses GIL limitation Employing a multi-process pipeline architecture, each worker process has an independent Python interpreter, completely bypassing the Global Interpreter Lock (GIL). Experiments have verified that under heavy load scenarios (3000 iterations), the serial processing throughput is 13.9 fps, while the four-process pipeline of this invention achieves a throughput of 30.6 fps, a speedup of 2.20 times, and a 120% improvement in processing efficiency. This solves the technical challenge of Python multithreading being unable to truly execute CPU-intensive tasks in parallel.

[0024] 2. Batch transmission mechanism reduces IPC overhead by more than 95%. By adopting a batch data transmission mechanism, the number of inter-process communications is reduced from 2N times to 0.1N times. When the batch size is 20 frames, the number of communications is reduced to 1 / 20 of that of single-frame transmission, and the communication overhead is reduced from 30%-50% to 1%-3%, breaking through the technical prejudice in this field that "multi-process is not suitable for real-time processing".

[0025] 3. Work stealing scheduling achieves dynamic load balancing. Each worker process maintains an independent double-ended queue. Local processes acquire tasks from the head of the queue (LIFO), while idle processes steal tasks from the tail of other queues (FIFO). This mechanism increases multi-core utilization from 62% to over 85%.

[0026] 4. Load-aware scheduling adaptively adjusts parallel granularity The number of processes is dynamically adjusted based on serial processing efficiency. When efficiency is >60%, reduce processes to avoid communication overhead; when efficiency is <30%, enable all processes. Under light load (98.4% efficiency), a single process is optimal, while under heavy load (9.3% efficiency), four processes achieve a 2.20x speedup.

[0027] 5. The experimental verification is sufficient and the statistical significance is extremely high. Independent samples t-test analysis showed that the throughput difference between this invention and serial processing under heavy load scenarios was statistically significant. p = 6.33 × 10 -12 <0.001 Cohen's d = 42.66 (large effect); 95% confidence intervals have no overlap; The above results show that the present invention achieves a speedup of 2.20 times under heavy computing scenarios, which is statistically significant and far superior to the traditional serial scheme (1.0x) and the Python multithreaded scheme (<1.2x), proving that the performance improvement of the present invention has high statistical reliability.

[0028] 6. Developed entirely in Python, with convenient cross-platform deployment. Implemented entirely in Python, it requires no C language extensions or cross-compilation, offering excellent cross-platform compatibility and ease of deployment. It natively adapts to ARM embedded hardware, making development and deployment relatively simple. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall system architecture of the present invention, showing the cascading relationship of each module and the data flow; Figure 2 This is a comparative diagram of single-frame and batch data transmission; Figure 3 This is a schematic diagram of the complete process of load balancing scheduling achieved by stealing work data in this invention; Figure 4 This is a pseudocode diagram of an algorithm that uses work-stealing to achieve load balancing scheduling. Figure 5 This is a graph showing the relationship between the computational load and the parallel speedup ratio of the system of this invention. Figure 6 This is a schematic diagram summarizing the experimental data, showing a bar chart comparing the throughput and speedup ratio of the serial scheme and the present invention under three sets of gradient loads. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0031] The real-time spectrum data stream processing system based on multi-process pipeline of the present invention, such as Figure 1 As shown, it includes a data acquisition module, a batch packaging module, a task queue module, a calculation module, a job stealing scheduler, a result collection module, and a load-aware scheduler.

[0032] The data acquisition module's input interface connects to an external spectrum data source to read IQ sampled data or raw spectrum data and outputs a single-frame data stream. The data source can be an analog high-speed data stream, TinySA Ultra hardware acquisition, or IQ sampled data. The data acquisition module's output interface sends a single-frame data stream to the batch packaging module.

[0033] The input of the batch packaging module is connected to the output of the data acquisition module. It has an internal data buffer to temporarily store multiple received single-frame data. When the number of data frames in the buffer reaches a preset number, the entire batch of data is packaged into a batch data packet. The output of the batch packaging module sends the batch data packet to the task queue module.

[0034] The input of the task queue module is connected to the output of the batch packaging module. It uses a multiprocessing safe queue (multiprocessing.Queue) with a preset capacity of 200 to cache batch data packets and decouple the data production and calculation processes. Its output is connected to the input of the calculation module.

[0035] The input of the computation module is connected to the output of the task queue module, and it contains several worker processes. The total number of worker processes is dynamically determined based on the number of CPU cores, taking the value as the number of physical CPU cores minus one, with one core reserved for operating system scheduling. For example, a Raspberry Pi has 4 CPU cores, so the number of worker processes is 3. Each worker process has its own independent Python interpreter and memory space, and each worker process can independently execute FFT transformation, peak detection, noise basis calculation, and spectrum analysis tasks. The output of each worker process in the computation module sends the processing results to the result collection module.

[0036] The computing module supports three computing modes: light computing mode, which only performs peak detection; medium computing mode, which performs FFT transformation; and heavy computing mode, which performs FFT transformation, multi-peak detection, noise floor calculation, and spectral feature extraction. The load-aware scheduler switches the computing module's operating mode according to the load density.

[0037] The work-stealing scheduler establishes a control connection with all worker processes to monitor the task queue storage of each process in real time. When a worker process's local queue is detected to be empty, tasks are stolen from the tail of other worker process queues and assigned to that idle process, achieving dynamic load balancing among the worker processes. The work-stealing scheduler is configured with dual-ended task queue scheduling logic: local processes read tasks from the head of the queue, while idle processes steal tasks from the tail of the queue, and a random round-robin mechanism is used to select the target queue for stealing.

[0038] The input of the results collection module is connected to the output of all working processes. It is used to collect the processing results of each working process, summarize the spectrum processing results, and calculate performance indicators such as total number of processed frames, average throughput, and speedup ratio, and generate a JSON format running log file.

[0039] The load-aware scheduler's input is connected to the data acquisition module's output, acquiring data on the data source's output rate and serial processing frame rate. It then outputs control signals to the computing module, dynamically adjusting the number of active worker processes within the computing module based on the serial processing efficiency. The load-aware scheduler's control terminal is connected to the computing module, used to monitor the computing load density in real time and dynamically adjust the number of active worker processes based on the serial processing efficiency. The load-aware scheduler adjusts the number of active worker processes using the following rule: Serial processing efficiency = (Serial processing output frame rate ÷ Data source output frame rate) × 100%, with a first threshold of 60% and a second threshold of 30%. The monitoring period is 1000 milliseconds, and adjustment is performed when the threshold conditions are met for two consecutive monitoring periods. When the efficiency is higher than 60%, the number of worker processes is reduced to avoid communication overhead; when the efficiency is lower than 30%, all worker processes are enabled.

[0040] The present invention provides a real-time spectrum data stream processing method based on a multi-process pipeline, characterized by comprising the following steps: S1. Construct a three-stage pipelined processing architecture consisting of a data acquisition stage, an FFT computation stage, and a spectrum post-processing stage. The FFT computation stage is configured with several worker processes, the total number of which is equal to the number of physical CPU cores minus one. Each worker process has an independent Python interpreter and memory space, independently executing FFT transform and spectrum analysis tasks, thereby bypassing the Global Interpreter Lock (GIL) limitation. Data exchange between the pipeline stages is completed through process-safe queues.

[0041] S2. Data Transmission: Pack multiple frames of spectrum data into batch data packets, and complete the unified transmission between processes in units of batch data packets.

[0042] A batch data transmission mechanism is adopted to package multiple data frames and transmit them to the working process at once. The number of data frames encapsulated in the batch data packet is adaptively adjusted according to the data source output rate: when the data source output rate is higher than 200 frames / second, 30 to 50 frames are encapsulated in a single batch; when the data source output rate is lower than 100 frames / second, 10 frames are encapsulated in a single batch.

[0043] A single batch of data packets completes two inter-process communications: one transmission and one result return. When 20 frames are used as the single batch encapsulation standard, the total number of inter-process communications for N frames of data is 0.1N.

[0044] The number of inter-process communication (IPC) communications is reduced from 2N times to 0.1N times. Communication overhead is reduced by more than 95%. The number of data frames per batch is dynamically adjusted according to the data source rate: the batch size is increased to 30-50 frames at high data rates and decreased to 10 frames at low data rates.

[0045] In traditional single-frame transmission, each frame of data requires two inter-process communication (IPC) operations: the first to transfer data from the producer process to the consumer process, and the second to return the processing result from the consumer process to the producer process. For N frames of data, this results in a total of 2N IPC operations. Real-world testing shows that in single-frame transmission mode, IPC overhead accounts for as much as 30%-50%, severely limiting system throughput. When the data source rate is high, communication overhead becomes the main performance bottleneck.

[0046] In the batch transmission method of this invention, 20 frames of data can be packaged into a batch and transmitted to the working process at once. After the working process processes the entire batch, it returns the batch results all at once.

[0047] like Figure 2 As shown, the batch transmission mechanism is illustrated using 20 frames per batch as an example: Number of communications per batch: 2 IPCs (one batch send + one batch return); Total number of communications for N frames: 2 × (N / 20) = 0.1N IPCs; Communication overhead is reduced by 95% (from 2N times to 0.1N times); The proportion of communication expenses has decreased from 30%-50% to 1%-3%.

[0048] The relationship between batch size and system performance can be described by the following formula: Throughput formula: T=1 / [Tacq+Tcompute / N+Tipc×2 / B], Delay formula: L = B / R + Tcompute / N + Tipc Where Tacq is the single-frame acquisition time, Tcompute is the single-frame computation time, Tipc is the single IPC overhead, B is the batch size (frames / batch), and R is the data source rate (fps).

[0049] From the above formula, it can be seen that, Increasing batch size B leads to a decrease in IPC frequency, which in turn increases throughput T (but there is a marginal effect). Batch size B increases → waiting time B / R increases → delay L increases.

[0050] The dynamic adjustment strategy aims to achieve an optimal balance between throughput and latency based on the data source rate R.

[0051] Batch size can be dynamically adjusted according to the data source rate: When the data source rate is high (e.g., 300fps), the frame interval is short, and increasing the batch size has a relatively limited impact on end-to-end latency, but can significantly reduce the number of inter-process communications. Therefore, the system automatically increases the batch size to 30-50 frames to reduce communication overhead and improve throughput.

[0052] When the data source rate is low (e.g., 50fps), the frame interval is long, and the communication overhead is relatively small. Reducing the batch size can effectively reduce data waiting latency and ensure the system's real-time response capability. Therefore, the system automatically reduces the batch size to 10-15 frames.

[0053] S3. Each worker process is configured with an independent task queue. Idle worker processes grab tasks to be processed from the tail of the task queues of other worker processes, thus evenly distributing the computing load of each process.

[0054] Specifically, each worker process maintains an independent double-ended task queue. The local worker process reads tasks from the head of the queue, while the idle process that steals tasks reads tasks from the tail of the queue. A random polling mechanism is used to select the queue of tasks to be stolen.

[0055] The random polling mechanism works as follows: after removing the currently idle process, the numbers of all remaining working processes are randomly shuffled, and tasks are attempted to be stolen in the shuffled order. The polling stops when a valid task is obtained, until the entire task queue has been traversed.

[0056] like Figure 3 and Figure 4 As shown, the local process retrieves tasks from the head of the queue (LIFO strategy), leveraging cache locality to improve efficiency. When a process's local queue is empty, that process becomes a stealer, retrieving tasks from the tail of other processes' queues (FIFO strategy), reducing lock contention. The target queue for stealing is selected using a random round-robin method to prevent multiple stealers from competing for the same queue simultaneously.

[0057] The stealing operation is protected by a mutex lock, but the stealing frequency is much lower than the local task acquisition frequency, resulting in a low probability of lock contention. Testing showed that the job-stealing scheduler increased multi-core utilization from 62% to over 85%.

[0058] S4. Monitor the serial processing efficiency of the system in real time and dynamically adjust the number of active working processes within the FFT computing level based on the efficiency threshold. When the serial processing efficiency is higher than the first threshold, it is determined to be a light load and the number of active working processes is reduced. When the serial processing efficiency is lower than the second threshold, it is determined to be a heavy load and the number of active working processes is increased to the preset upper limit (number of CPU cores - 1).

[0059] The load-aware scheduler monitors the serial processing efficiency in real time. The formula for calculating the serial processing efficiency is: Efficiency = Serial processing frame rate / Data source frame rate * 100%, with a first threshold of 60% and a second threshold of 30%.

[0060] When efficiency exceeds 60%, it is considered a light load scenario. In this case, the computational task is relatively light, and multi-process communication overhead is dominant. The load-aware scheduler reduces the number of active worker processes to 1-2. When efficiency is between 30% and 60%, it is considered a medium load scenario, and the load-aware scheduler activates 2-3 worker processes. When efficiency is below 30%, it is considered a heavy load scenario, where the computational task becomes the main bottleneck, and the load-aware scheduler activates all worker processes (CPU core count - 1).

[0061] The system serial processing efficiency monitoring period is 1000 milliseconds. When the threshold condition is met for two consecutive monitoring periods, the number of working processes is adjusted.

[0062] Verification experiment: Verification experiments were run on a Raspberry Pi 5 platform (4-core ARM Cortex-A76@2.4GHz, 8GB RAM).

[0063] Experimental setup: Test Scenario 1 (Light Load): Data rate 100fps, number of iterations 500, number of repetitions 5.

[0064] Test Scenario 2 (Medium Load): Data rate 150fps, number of iterations 1500, number of repetitions 5.

[0065] Test Scenario 3 (Heavy Load): Data rate 150fps, number of iterations 3000, number of repetitions 5.

[0066] The experimental results are shown in the table below:

[0067] Statistical conclusions: When the computational load is light (500 iterations), serial processing has reached the upper limit of the data source (98.4fps vs 100fps), and multi-process communication overhead dominates, so this invention has no advantage. When the computational load is moderate (1500 iterations), serial and multi-process performance are comparable, at a balance point (29.1 vs 31.1 fps). When computational load becomes the main bottleneck (3000 iterations), computation becomes the main bottleneck, and the advantages of multi-process are significant. The serial speed drops to 13.9fps, while the present invention is stable at 30.6fps, with a speedup of 2.20 times.

[0068] Independent samples t-tests showed that the performance difference between this invention and serial processing was statistically significant. t-statistic: 60.33; p = 6.33 × 10 -12 <0.001; Cohen's d = 42.66 (large effect); The 95% confidence intervals are non-overlapping.

[0069] The above results show that the present invention achieves a speedup of 2.20 times under heavy computing scenarios, which is statistically significant and far superior to the traditional serial scheme (1.0x) and the Python multithreaded scheme (<1.2x). This proves that the performance improvement of the present invention has high statistical reliability and is not a coincidence.

[0070] like Figure 5 As shown, there is a positive correlation between computational load and speedup: the heavier the computational load, the more significant the parallel acceleration effect. In heavy-load scenarios, this invention achieves a 2.20x speedup and a 120% improvement in single-frame processing efficiency.

[0071] like Figure 6 As shown in the bar chart summarizing the experimental data, the performance advantages of this invention under different load scenarios are intuitively demonstrated.

[0072] Real hardware verification: The above system was integrated with the TinySA Ultra spectrum analyzer as a real hardware verification.

[0073] The TinySA Ultra is connected to the Raspberry Pi 5 via USB, and the data acquisition module reads spectrum data from the TinySA Ultra. Verification results show that the system runs stably, functions correctly, and can process the spectrum data acquired by the TinySA Ultra in real time. The data acquisition module can correctly identify the TinySA Ultra's serial port device ( / dev / ttyACM1) and acquire spectrum data at approximately 30fps. The calculation module can perform FFT transformation and peak detection in real time.

[0074] The following key code examples for core modules are only used to illustrate the scheduling logic and are not intended to limit the scope of protection: def worker_process(worker_id, task_queue, result_queue): "Worker process main loop: Local acquisition + Work theft" local_queue = deque() while running: # Prioritize retrieving from the local queue task = local_queue.popleft() if local_queue else None if task is None: # Random polling to select victims victims = list(range(num_workers)) random.shuffle(victims) for i in victims: if i != worker_id: task = steal_queues[i].pop_back() if task: break if task: result = fft_compute(task) result_queue.put(result) The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time spectrum data stream processing method based on a multi-process pipeline, characterized in that, Includes the following steps: S1. Build a three-stage pipelined processing architecture consisting of a data acquisition stage, an FFT calculation stage, and a spectrum post-processing stage. The FFT computation stage is configured with several worker processes, each with an independent Python interpreter and independent memory space, to independently perform FFT transformation and spectrum analysis operations; data exchange between each stage of the pipeline is completed through a process-safe queue. S2. Pack multiple frames of spectrum data into batch data packets, and complete the unified transmission between processes in units of batch data packets; S3. Each worker process is configured with an independent task queue. Idle worker processes grab tasks to be processed from the tail of the task queues of other worker processes to evenly distribute the computing load of each process. S4. Monitor the serial processing efficiency of the system in real time and dynamically adjust the number of active worker processes in the FFT computing stage according to the efficiency threshold. When the serial processing efficiency is higher than the first threshold, it is judged as a light load and the number of active worker processes is reduced. When the serial processing efficiency is lower than the second threshold, it is determined to be a heavy load, and the active working processes are expanded to the preset limit.

2. The real-time spectrum data stream processing method based on multi-process pipeline according to claim 1, characterized in that: In step S1, the total number of working processes is taken as the number of physical CPU cores minus 1. The processor is a quad-core ARM processor, which is configured with 3 working processes.

3. The real-time spectrum data stream processing method based on multi-process pipeline according to claim 1, characterized in that: In step S2, the number of data frames encapsulated in the batch data packets is adaptively adjusted according to the data source output rate: when the data source output rate is higher than 200 frames / second, 30 to 50 frames are encapsulated in a single batch; when the data source output rate is lower than 100 frames / second, 10 frames are encapsulated in a single batch.

4. The real-time spectrum data stream processing method based on multi-process pipeline according to claim 1, characterized in that: In step S2, a single batch of data packets completes one transmission and one result return, totaling two inter-process communications. When 20 frames are used as the single batch encapsulation standard, the total number of inter-process communications for N frames of data is 0.1N.

5. The real-time spectrum data stream processing method based on multi-process pipeline according to claim 1, characterized in that: In step S3, each worker process is configured with a dual-end task queue. The local worker process reads tasks from the head of the queue, and the idle process that steals tasks reads tasks from the tail of the queue. A random polling mechanism is used to select the queue of tasks to be stolen. The random polling mechanism is as follows: after removing the currently idle process, the numbers of all remaining working processes are randomly shuffled, and tasks are attempted to be stolen in the shuffled order. The polling stops when a valid task is obtained, until the entire task queue has been traversed.

6. The real-time spectrum data stream processing method based on multi-process pipeline according to claim 1, characterized in that: In step S4, the serial processing efficiency calculation formula is: serial processing efficiency = (serial processing output frame rate ÷ data source output frame rate) × 100%; the first threshold is set to 60%, and the second threshold is set to 30%.

7. The real-time spectrum data stream processing method based on multi-process pipeline according to claim 6, characterized in that: The system serial processing efficiency monitoring period is 1000 milliseconds. When the threshold condition is met for two consecutive monitoring periods, the number of working processes is adjusted.

8. A real-time spectrum data stream processing system based on a multi-process pipeline, characterized in that, include: The data acquisition module connects to an external spectrum data source at its input end to read IQ sampling data or raw spectrum data and outputs a single-frame data stream. The batch packaging module connects to the output of the data acquisition module at its input end. It has a built-in data buffer that caches single-frame data streams. Once the number of data frames in the buffer reaches a preset number, it is packaged into batch data packets and output. The task queue module uses a multi-process safe queue to cache batch data packets, thereby decoupling the data production and computation processes. The computation module contains several worker processes, each configured with an independent Python interpreter and independent memory space, and independently performs FFT transformation and spectrum analysis operations. The job stealing scheduler establishes a control connection with all worker processes, monitors the task queue storage of each process in real time, and when it detects that the worker process queue is empty, it grabs tasks from the tail of other process queues and assigns them to the idle process. The results collection module connects to the output of all working processes, summarizes the spectrum processing results, statistically analyzes system performance indicators, and generates operation logs. The load-aware scheduler collects data on the data source output rate and serial processing frame rate, outputs control signals to the computing module, and dynamically adjusts the number of active working processes within the computing module based on the serial processing efficiency.

9. The real-time spectrum data stream processing system based on a multi-process pipeline according to claim 8, characterized in that: The computing module is divided into three computing modes: light mode only performs signal peak detection; medium mode performs FFT transformation operation; and heavy mode simultaneously completes FFT transformation, multi-peak identification, noise basis solution and spectral feature extraction. The load-aware scheduler switches the corresponding working mode according to the real-time calculated load density.

10. The real-time spectrum data stream processing system based on a multi-process pipeline according to claim 8, characterized in that: The job stealing scheduler is configured with dual-end task queue scheduling logic. Local processes read tasks from the head of the queue, while idle processes steal tasks from the tail of the queue, and a random polling mechanism is used to select the target queue for stealing.

11. The real-time spectrum data stream processing system based on a multi-process pipeline according to claim 8, characterized in that: The load-aware scheduler adjusts the number of active worker processes using the following rules: serial processing efficiency = (serial processing output frame rate ÷ data source output frame rate) × 100%, with a first threshold of 60% and a second threshold of 30%. The monitoring period is 1000 milliseconds, and the adjustment is performed when the threshold conditions are met for two consecutive monitoring periods.

Images