A parallel method and device for accelerating the search pipeline of pulsar Fourier domain

Abstract

The present invention discloses a parallel method and device for accelerating the search pipeline of pulsar Fourier domain, including: allocating received astronomical data to multiple parallel processes; each process includes three serial sub-threads, using the first CPU sub-thread to pre-process the allocated astronomical data, using the GPU sub-thread to read the pre-processed data and perform accelerated calculation on the GPU to obtain candidate signal data, using the second CPU sub-thread to read the candidate signal data and perform post-processing and result aggregation, while using queues and queue blocking locks of multiple parallel process architectures to synchronize the task status between sub-threads; dynamically adjusting the number of processes in the processing flow through monitoring feedback; recording the task status and the calculation results of each process in real time and handling abnormal situations. The present invention can effectively improve the pulsar binary search process, and support the use of multi-GPU parallel acceleration, greatly improving the speed of searching for this type of celestial body from FAST astronomical data.

Description

Technical Field

[0001] The present invention belongs to the technical field of high-performance computing of astronomical data, and in particular relates to a parallel method and device for accelerating the search pipeline of a pulsar in the Fourier domain. Background Art

[0002] Pulsars are dense neutron stars in the universe that have high-speed rotation and strong electromagnetic radiation characteristics. They have become important targets in astronomical observations due to their stable rotation period. The discovery of pulsars has provided rich data for cosmology and astrophysics, helping scientists to gain a deeper understanding of key physical phenomena such as gravitational waves and general relativity. However, with the continuous advancement of observation technology, especially the commissioning of high-sensitivity telescopes such as FAST (Five-hundred-meter Aperture Spherical Radio Telescope), the amount of data generated has increased dramatically. Pulsar searches require processing massive amounts of observational data, which has brought unprecedented challenges. Although traditional time-domain search methods can identify the signals of single pulsars to a certain extent, they are difficult to cope with the complexity of binary star systems and the need for rapid searches. Therefore, it is crucial to develop new search methods.

[0003] In order to meet the above challenges, the Fourier Domain Accelerated Search (FDAS) algorithm came into being. This algorithm significantly improves the efficiency of pulsar searches by converting observational data into the frequency domain for processing and using mathematical tools such as the Fast Fourier Transform (FFT). Currently, the Presto project, as one of the representative solutions of the FDAS algorithm, has demonstrated its powerful search capabilities in practice. However, its implementation on the GPU has significant performance bottlenecks. The main problem is that the existing command-line-based multi-process solutions are not efficient on the GPU. Due to the inefficient allocation and management of GPU resources, the GPU utilization rate is low, the computing delay is significantly increased, and the overall system throughput cannot meet expectations. At the same time, due to large performance fluctuations and overly concentrated resource consumption, the search process not only consumes a large amount of computing resources, but also brings high operating costs. These problems severely limit the application potential of the Presto project and similar solutions in large-scale astronomical data processing.

[0004] With the rapid development of heterogeneous computing technology, especially the popularization of CPU and GPU collaborative computing mode, new solutions have been provided for computationally intensive tasks such as pulsar search. However, the existing FDAS algorithm has not yet fully utilized its performance advantages. The current technology does not fully combine the application layer characteristics and the advantages of the underlying hardware architecture in design, resulting in the system performance not being maximized in actual operation, and there is still much room for improvement in computing efficiency and resource utilization.

[0005] Therefore, in order to overcome the performance bottleneck problem in the existing scheme, fully tap the potential of heterogeneous computing architecture, and achieve more efficient and economical pulsar search, it is necessary to deeply redesign the FDAS algorithm, optimize the data processing flow, and improve the parallelism and execution efficiency of the algorithm. This will not only promote the in-depth development of pulsar research, but also make important contributions to the progress of cosmology and astrophysics. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a parallel method and device for accelerating the search pipeline of pulsars in the Fourier domain. By designing a multi-threaded architecture in multiple processes and in each process to process astronomical data in parallel, the pulsar binary search process can be improved with several times the performance advantage. At the same time, it supports the use of multi-GPU parallel acceleration, which can increase the speed of searching for this type of celestial body from FAST astronomical data by dozens of times.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a pulsar Fourier domain accelerated search pipeline parallel method, comprising the following steps:

[0009] Distribute received astronomical data to a configurable number of parallel processes;

[0010] Each process is divided into three serial sub-threads. The first CPU sub-thread is used to pre-process the assigned astronomical data. The GPU sub-thread is used to read the pre-processed data and perform accelerated calculations on the GPU to obtain candidate signal data. The second CPU sub-thread is used to read the candidate signal data and perform post-processing and result aggregation. At the same time, queues and queue blocking locks of multiple parallel process architectures are used to synchronize the task status between sub-threads.

[0011] Dynamically adjust the number of processes in the processing flow through monitoring feedback;

[0012] Record task status and calculation results of multiple parallel processes in real time and handle exceptions.

[0013] Specifically, the preprocessing of the allocated astronomical data by the first CPU subthread includes:

[0014] In the first CPU sub-thread, the allocated astronomical data is preprocessed including creating and initializing harmonic and sub-harmonic information structures, wherein the harmonic and sub-harmonic information structures include frequency domain data distribution of each harmonic and its memory requirement.

[0015] Specifically, the method of using a GPU sub-thread to read the pre-processed data and performing accelerated calculation on the GPU to obtain candidate signal data includes:

[0016] In the GPU sub-thread, the harmonic and sub-harmonic information structure obtained by preprocessing of the first CPU sub-thread is read, and according to the predefined task parameters, resources are allocated on the GPU and accelerated calculations including Fourier transform and candidate signal data generation are performed to obtain candidate signal data.

[0017] Specifically, the method of using the second CPU sub-thread to read the candidate signal data and perform post-processing and result aggregation includes:

[0018] In the second CPU sub-thread, the candidate signal data obtained by the accelerated calculation of the GPU sub-thread is read, and the candidate signal data is post-processed including sorting and screening and formatted and output to obtain a result summary file.

[0019] Specifically, the predefined task parameters include:

[0020] The maximum z value of the accelerated search, the maximum w value of the accelerated search, the threshold of signal detection, the number of harmonics used in the accelerated search, and the amount of data processed in one task, where the z value represents the width of the Fourier window and the w value represents the accelerated search depth parameter.

[0021] Specifically, the task status includes:

[0022] The task ID, current processing stage, executing process ID, task start time and last activity time, and task completion timestamp or timeout status.

[0023] Specifically, dynamically adjusting the number of processes in the processing flow through monitoring feedback includes:

[0024] When a task execution timeout is detected, the timed-out task is reallocated to the task queue;

[0025] The original process where the timed-out task is located is processed, including: setting the stop flag of the process to terminate the original process, monitoring the exit status of the process and waiting for the process to exit normally within the set time, forcibly terminating the process if the process does not exit within the predetermined time, and recreating a new process after the original process exits and adding the new process to the working process pool to ensure that the number of processes in the working process pool reaches the specified number.

[0026] Specifically, the method further includes:

[0027] Provides debugging mode and non-debugging mode selection. In non-debugging mode, standard output is redirected to remove redundant task execution log records. In debugging mode, detailed task execution logs are retained and monitored and troubleshooted based on the detailed task execution logs.

[0028] In a second aspect, to achieve the above-mentioned invention object, an embodiment of the present invention further provides a pulsar Fourier domain accelerated search pipeline parallel device, which is implemented using the above-mentioned pulsar Fourier domain accelerated search pipeline parallel method, including: a task allocation module, a task execution module, a task monitoring module and a task recording module;

[0029] The task allocation module is used to allocate the received astronomical data to a configurable number of multiple parallel processes;

[0030] The task execution module is used to divide each process into three serial sub-threads, use the first CPU sub-thread to pre-process the assigned astronomical data, use the GPU sub-thread to read the pre-processed data and perform accelerated calculation on the GPU to obtain candidate signal data, use the second CPU sub-thread to read the candidate signal data and perform post-processing and result aggregation, and use the queues and queue blocking locks of the multiple parallel process architectures to synchronize the task status between the sub-threads;

[0031] The task monitoring module is used to dynamically adjust the number of processes in the processing flow through monitoring feedback;

[0032] The task recording module is used to record the task status and the calculation results of multiple parallel processes in real time and handle abnormal situations.

[0033] In the third aspect, to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides an electronic device, comprising a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned pulsar Fourier domain accelerated search pipeline parallel method when executing the computer program.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) Isolation is achieved through multi-process parallel processing, so that each process has independent memory space and resources, which effectively avoids thread conflicts between multiple processes and ensures thread safety. This isolation mechanism not only improves the stability of the program, but also provides a reliable operating environment for complex data processing.

[0036] (2) By setting up multiple serial sub-threads in each process and cleverly utilizing the pipeline working method, the GPU resources are utilized without waiting. This method enables the GPU to continuously receive processing tasks, thereby greatly improving the GPU resource utilization rate. In addition, the pipeline method also optimizes task allocation and scheduling, further improving the overall processing efficiency.

[0037] (3) Due to the complexity of astronomical data structures, complex data processing and exchange only through multiple processes will lead to shared memory problems. However, the multi-process provided by the present invention and the use of multiple sub-threads in each process not only fully utilize the parallel computing capabilities of multi-core processors, but also effectively avoid the shared memory problem through reasonable task division and inter-thread communication mechanism, thereby significantly improving data processing efficiency, reducing system overhead, and enhancing program stability and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 It is a flow chart of a parallel method for accelerating the search pipeline of pulsar Fourier domain provided by an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of a sub-process workflow in each process provided by an embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of a process for processing an original process of a timed-out task provided by an embodiment of the present invention;

[0042] Figure 4 It is a structural schematic diagram of a parallel device for accelerating the search pipeline of a pulsar in Fourier domain provided by an embodiment of the present invention;

[0043] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0045] The inventive concept of the present invention is: in response to the problems of low parallelism and execution efficiency of pulsar Fourier domain accelerated search algorithms in the prior art, the embodiments of the present invention provide a pulsar Fourier domain accelerated search pipeline parallel method and device, which processes astronomical data by designing multiple processes, and designs a multi-threaded processing stage including a first CPU sub-thread, a GPU sub-thread and a second CPU sub-thread in each process, which can accelerate the parallel processing efficiency of astronomical data, effectively improve the pulsar binary search process, and support the use of multi-GPU parallel acceleration, which can increase the speed of searching for this type of celestial body from FAST astronomical data by dozens of times.

[0046] Figure 1 FIG. 1 is a flow chart of a parallel method for accelerating the search pipeline of a pulsar in the Fourier domain provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a pulsar Fourier domain accelerated search pipeline parallel method, comprising the following steps:

[0047] S1 distributes the received astronomical data to a configurable number of parallel processes.

[0048] In an embodiment, astronomical data is received, which is an FFT (Fast Fourier Transform) file generated by the realfft command in the pulsar search and analysis software developed by Scott Ransom. The FFT file contains frequency information of the astronomical observation signal, specifically the Fourier transform result based on the sampling points.

[0049] Initialize a task queue for receiving and managing FFT files. Initialize a result queue for receiving the final processing results. The result queue receives the processing results generated by the second CPU subthread after each task is completed, including the output data of the accelerated search and its related statistical information. Initialize a worker process pool consisting of multiple worker processes, each of which contains three subthreads and two intermediate result queues, and create a specified number of worker threads for each GPU to ensure that the system can efficiently process FFT files from different sources.

[0050] Traverse the file directory in the FFT file, generate an independent task for each file and assign it to the global task queue for processing, initialize multiple processing processes to obtain different tasks from the global task queue for processing.

[0051] S2, divides each process into three serial sub-threads, uses the first CPU sub-thread to pre-process the assigned astronomical data, uses the GPU sub-thread to read the pre-processed data and perform accelerated calculations on the GPU to obtain candidate signal data, uses the second CPU sub-thread to read the candidate signal data and perform post-processing and result aggregation, and simultaneously uses queues and queue blocking locks of multiple parallel process architectures to synchronize the task status between sub-threads.

[0052] In the embodiment, Figure 2 As shown, in the first CPU subthread, tasks are extracted from the task queue, and the assigned astronomical data is preprocessed, including creating and initializing harmonic and subharmonic information structures, wherein the harmonic and subharmonic information structures include the frequency domain data distribution of each harmonic and its memory requirements, so as to prepare for the subsequent efficient FFT transformation on the GPU. Harmonic refers to the frequency component that appears at an integer multiple of the original signal frequency. They are usually extracted from the signal by Fourier transform or other spectrum analysis methods, and are used to enhance or filter out signal components of a specific frequency. Sub-harmonic refers to the frequency component that is lower than the fundamental harmonic, which is usually a fractional multiple of the fundamental frequency (such as 1 / 2, 1 / 3, etc.). Sub-harmonics are used in some signal processing to more finely decompose the spectral information of the signal to support multi-level frequency analysis. The harmonic and sub-harmonic information structures are passed to the GPU subthread through the first intermediate result queue to continue the calculation.

[0053] In the embodiment, Figure 2 As shown, in the GPU sub-thread, the harmonic and sub-harmonic information structure is obtained through the first intermediate result queue, and a specific GPU device is allocated according to the environment variable CUDA_VISIBLE_DEVICES. According to the predefined task parameters, specific resources are allocated on the GPU and accelerated calculations including Fourier transform and candidate signal data generation are performed to form a linked list structure of candidate signal data. GPU acceleration greatly shortens the processing time and improves the overall efficiency of the search. Finally, the candidate signal data is passed to the second CPU sub-thread through the second intermediate result queue. Among them, the predefined task parameters include:

[0054] (1) zmax: The maximum z value for accelerated search. The z value represents the width of the Fourier window, controls the frequency resolution, and has a direct impact on signal detection accuracy.

[0055] (2) wmax: the maximum w value of the accelerated search. The w value represents the accelerated search depth parameter and controls the acceleration range of the signal in the Fourier domain. The higher the z value, the wider the signal acceleration range that can be detected.

[0056] (3) sigma: threshold for signal detection;

[0057] (4) numharm: the number of harmonics used in the acceleration search;

[0058] (5) batchsize: the amount of data processed in one task.

[0059] In the embodiment, Figure 2 As shown, in the second CPU sub-thread, the candidate signal data in the linked list structure is obtained through the second intermediate result queue, and post-processing including sorting and screening is performed and formatted and output as a CSV file containing detailed information of the candidate pulsar signals to the result queue. After all process calculations are completed, the processing of the entire pulsar search task is completed.

[0060] At the same time, the queue and queue blocking lock of the multi-process architecture are used to synchronize the task status between each sub-thread. The task status includes:

[0061] (1) Task ID;

[0062] (2) Current processing stage;

[0063] (3) The ID of the executed process;

[0064] (4) The start time and last active time of the task;

[0065] (5) The timestamp or timeout status of the task completion.

[0066] S3, dynamically adjusts the number of processes in the processing flow through monitoring feedback.

[0067] In the embodiment, when a task execution timeout is detected, the timed-out task is reallocated to the task queue. Figure 3 As shown, the original process where the timed-out task is located is processed, including:

[0068] (1) Set the stop flag of the process to terminate the original process;

[0069] (2) Monitor the exit status of the process and wait for the process to exit normally within the set time;

[0070] (3) If the process does not exit within the scheduled time, the process is terminated forcibly;

[0071] (4) After the original process exits, a new process is created and added to the working process pool to ensure that the number of processes in the working process pool reaches the specified number, thereby ensuring the continuity and stability of task processing and the dynamic recovery and rational use of system resources.

[0072] Through the above monitoring process, dynamic allocation and load balancing of tasks are achieved to ensure optimal utilization of resources and efficient execution of tasks.

[0073] S4, records the task status and calculation results of multiple parallel processes in real time and handles abnormal situations.

[0074] In the embodiment, the execution status of each task is tracked and recorded, and the task execution status is written into the global task status dictionary to ensure that the results of task execution can be efficiently collected and returned. After all tasks are processed, the overall results will be output, including the processing time, status and final processing output of each file.

[0075] In addition, debug mode and non-debug mode options are provided. In non-debug mode, standard output is redirected to remove redundant task execution log records. In debug mode, detailed task execution logs are retained and monitored and troubleshooted based on the detailed task execution logs.

[0076] In summary, the embodiment of the present invention provides a pulsar Fourier domain accelerated search pipeline parallel method, which can improve the pulsar binary search process with several times the performance advantage, and at the same time supports the use of multi-GPU parallel acceleration, which can increase the speed of searching for this type of celestial body from FAST astronomical data by dozens of times.

[0077] Based on the same inventive concept, Figure 4 As shown, the embodiment of the present invention further provides a pulsar Fourier domain accelerated search pipeline parallel device 400, including: a task allocation module 410, a task execution module 420, a task monitoring module 430 and a task recording module 440.

[0078] The task allocation module 410 is used to allocate the received astronomical data to a configurable number of multiple parallel processes;

[0079] The task execution module 420 is used to divide each process into three serial sub-threads, use the first CPU sub-thread to pre-process the assigned astronomical data, use the GPU sub-thread to read the pre-processed data and perform accelerated calculation on the GPU to obtain candidate signal data, use the second CPU sub-thread to read the candidate signal data and perform post-processing and result aggregation, and use the queues and queue blocking locks of the multiple parallel process architectures to synchronize the task status between the sub-threads;

[0080] The task monitoring module 430 is used to dynamically adjust the number of processes in the processing flow through monitoring feedback;

[0081] The task recording module 440 is used to record the task status and the calculation results of multiple parallel processes in real time and handle abnormal situations.

[0082] Based on the same inventive concept, Figure 5As shown, an embodiment of the present invention further provides an electronic device 500, including a memory 510 and one or more processors 520, the memory 510 is used to store computer programs, and the processor 520 is used to implement the above-mentioned pulsar Fourier domain accelerated search pipeline parallel method when executing the computer program.

[0083] It should be noted that the above-mentioned embodiments provide a pulsar Fourier domain accelerated search pipeline parallel device and an electronic device, which belong to the same inventive concept as a pulsar Fourier domain accelerated search pipeline parallel method. The specific implementation process is detailed in the embodiment of a pulsar Fourier domain accelerated search pipeline parallel method, which will not be repeated here.

[0084] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A parallel pipeline method for accelerating the search of pulsars in Fourier domain, characterized in that: The following steps are involved: Distribute received astronomical data to a configurable number of parallel processes; Each process is divided into three serial sub-threads. The first CPU sub-thread is used to pre-process the assigned astronomical data. The GPU sub-thread is used to read the pre-processed data and perform accelerated calculations including Fourier transform and candidate signal data generation on the GPU to obtain candidate signal data. The second CPU sub-thread is used to read the candidate signal data and perform post-processing and result aggregation. At the same time, queues and queue blocking locks of multiple parallel process architectures are used to synchronize the task status between sub-threads. Dynamically adjust the number of processes in the processing flow through monitoring feedback, including: when a task execution timeout is detected, reallocate the timed-out task to the task queue; process the original process where the timed-out task is located, including: set the stop flag of the process to terminate the original process, monitor the exit status of the process and wait for the process to exit normally within the set time, forcibly terminate the process if the process does not exit within the predetermined time, recreate a new process after the original process exits and add the new process to the working process pool to ensure that the number of processes in the working process pool reaches the specified number; Record task status and calculation results of multiple parallel processes in real time and handle exceptions.

2. The pulsar Fourier domain accelerated search pipeline parallel method according to claim 1, characterized in that: The using the first CPU subthread to pre-process the allocated astronomical data includes: In the first CPU sub-thread, the allocated astronomical data is preprocessed including creating and initializing harmonic and sub-harmonic information structures, wherein the harmonic and sub-harmonic information structures include frequency domain data distribution of each harmonic and its memory requirement.

3. The pulsar Fourier domain accelerated search pipeline parallel method according to claim 2, characterized in that: The method of using the GPU sub-thread to read the pre-processed data and performing accelerated calculations including Fourier transform and candidate signal data generation on the GPU to obtain the candidate signal data includes: In the GPU sub-thread, the harmonic and sub-harmonic information structure obtained by preprocessing of the first CPU sub-thread is read, and according to the predefined task parameters, resources are allocated on the GPU and accelerated calculations including Fourier transform and candidate signal data generation are performed to obtain candidate signal data.

4. The pulsar Fourier domain accelerated search pipeline parallel method according to claim 1 or 3, characterized in that: The method of using the second CPU sub-thread to read the candidate signal data and perform post-processing and result aggregation includes: In the second CPU sub-thread, the candidate signal data obtained by the accelerated calculation of the GPU sub-thread is read, and the candidate signal data is post-processed including sorting and screening and formatted and output to obtain a result summary file.

5. The pulsar Fourier domain accelerated search pipeline parallel method according to claim 3, characterized in that: The predefined task parameters include: The maximum z value of the accelerated search, the maximum w value of the accelerated search, the threshold of signal detection, the number of harmonics used in the accelerated search, and the amount of data processed in one task, where the z value represents the width of the Fourier window and the w value represents the accelerated search depth parameter.

6. The pulsar Fourier domain accelerated search pipeline parallel method according to claim 1, characterized in that: Task statuses include: The task ID, current processing stage, executing process ID, task start time and last activity time, and task completion timestamp or timeout status.

7. The pulsar Fourier domain accelerated search pipeline parallel method according to claim 1, characterized in that: The method further comprises: Provides debugging mode and non-debugging mode options. In non-debugging mode, standard output is redirected to remove redundant task execution log records. In debugging mode, detailed task execution logs are retained and monitored and troubleshooted based on the detailed task execution logs.

8. A pulsar Fourier domain accelerated search pipeline parallel device, implemented by the pulsar Fourier domain accelerated search pipeline parallel method according to any one of claims 1 to 7, characterized in that: include: Task allocation module, task execution module, task monitoring module and task recording module; The task allocation module is used to allocate the received astronomical data to a configurable number of multiple parallel processes; The task execution module is used to divide each process into three serial sub-threads, use the first CPU sub-thread to pre-process the assigned astronomical data, use the GPU sub-thread to read the pre-processed data and perform accelerated calculation on the GPU to obtain candidate signal data, use the second CPU sub-thread to read the candidate signal data and perform post-processing and result aggregation, and use the queues and queue blocking locks of the multiple parallel process architectures to synchronize the task status between the sub-threads; The task monitoring module is used to dynamically adjust the number of processes in the processing flow through monitoring feedback; The task recording module is used to record the task status and the calculation results of multiple parallel processes in real time and handle abnormal situations.

9. An electronic device comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the pulsar Fourier domain accelerated search pipeline parallel method as described in any one of claims 1-7 when executing the computer program.

Images