Radio astronomy data parallel transmission method and device

Abstract

The present application relates to a kind of radio astronomy data parallel transmission method and device, method includes: main process traverses source directory, obtains the metadata of all files under source directory and file total number;Wherein, the metadata of each file includes the identification, size and source storage path of the file;Main process is according to the size of each file under source directory and file total number to each file grouping, obtains multiple file groups;Main process generates a transmission task for each file group, and write it into disk queue file to carry out persistent storage;Main process schedules multiple parallel running transmission sub-process, and each transmission sub-process executes the transmission task in disk queue file, each transmission sub-process executes a transmission task each time, to transmit the file group in the transmission task corresponding to each file from the source storage path of the file to target storage path.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and more specifically to a method and apparatus for parallel transmission of radio astronomy data. Background Technology

[0002] Radio astronomy observation has now entered a phase of explosive data growth. Taking the 110-meter QTT radio telescope, currently under construction, as an example, its wide frequency band coverage and high observation precision can generate terabytes of data in a single observation. This type of observation data is characterized by its large total data volume, highly heterogeneous file sizes, and massive number of files. Moreover, the QTT site has limited network resources. How to efficiently and reliably transmit massive amounts of heterogeneous observation data to the data center has become the core bottleneck of radio astronomy data management.

[0003] Currently, radio astronomy data transmission primarily employs traditional solutions such as remote file synchronization tools (rsync), parallel-optimized remote file synchronization tools, and SCP (Secure Copy Protocol), but all have significant technical drawbacks. Standard remote file synchronization tools rely on rolling checksums for incremental transmission, offering advantages in integrity verification. However, their serial transmission mode results in extremely low bandwidth utilization, and checksum calculations consume substantial CPU and memory resources, easily competing for resources with real-time scientific computing tasks at the observatory, leading to extremely poor transmission interruption recovery efficiency. While existing parallel-optimized remote file synchronization tools improve transmission throughput through multi-processing, they lack the ability to detect heterogeneous file sizes. In data scenarios dominated by extremely large files, the parallel acceleration effect fails, and they still suffer from high hardware resource overhead and the lack of a mechanism for resuming interrupted transmissions. Although the SCP protocol is simple to deploy and highly secure, it only supports full copying, resulting in significant transmission redundancy, and its serial transmission mode makes its overall transmission throughput insufficient to meet the demands of large-scale data observations.

[0004] Therefore, existing transmission schemes cannot adapt to the transmission scenarios of massive heterogeneous data in radio astronomy, and a new adaptive transmission scheme is urgently needed to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for parallel transmission of radio astronomy data, so as to solve the problems existing in the current transmission methods.

[0006] To achieve the above objectives, the present invention provides a method for parallel transmission of radio astronomy data, comprising:

[0007] The main process traverses the source directory to obtain the metadata of all files in the source directory and the total number of files; the metadata of each file includes the file's identifier, size, and source storage path;

[0008] The main process groups the files according to the size and total number of files in the source directory to obtain multiple file groups, each file group containing at least one file;

[0009] The main process generates a transfer task for each file group and writes each transfer task to a disk queue file for persistent storage; each transfer task includes a task identifier, the identifier of each file in the file group corresponding to the transfer task, the source storage path, and the target storage path;

[0010] The main process schedules multiple parallel-running transmission subprocesses, each of which executes a transmission task within a disk queue file. Each transmission subprocess executes one transmission task at a time to transfer each file in the filegroup corresponding to the transmission task from the source storage path to the target storage path.

[0011] Optionally, the main process groups the files according to their size and total number in the source directory, resulting in multiple file groups, specifically including:

[0012] The coefficient of variation is determined based on the size of each file in the source directory and the total number of files.

[0013] The correction factor is determined based on the coefficient of variation;

[0014] Files in the source directory whose size is greater than a preset size threshold are marked as oversized files, and all other files except the oversized files are marked as ordinary files. The number of oversized files in the source directory is counted.

[0015] The number of file groups is determined based on the correction factor, the preset size threshold, the total number of files in the source directory, and the number of oversized files in the source directory;

[0016] Each extremely large file is divided into a separate file group. All regular files are distributed to the remaining file groups after deducting the file groups corresponding to the extremely large files. During the distribution, the size of each regular file is used to balance the distribution so that the total amount of data in each remaining file group is similar, and the number of files in each file group does not exceed the upper limit of the number of files in a single group.

[0017] Optionally, the number of file groups satisfies the following relationship:

[0018] ,

[0019] in, It is the floor function. This is the maximum number of files in a single group. For the preset size threshold, This represents the number of extremely large files in the source directory. This represents the number of file groups. The correction factor is N, where N is the total number of files in the source directory and S is the total data volume of each file in the source directory.

[0020] Optionally, the main process schedules multiple parallel-running transmission subprocesses, each of which executes a transmission task within a disk queue file. Each transmission subprocess executes one transmission task at a time, transferring each file in the filegroup corresponding to that transmission task from its source storage path to its target storage path. Specifically, this includes:

[0021] The main process marks the initial state of all transmission tasks as the pending execution state and writes the state of all transmission tasks into the persistent state database.

[0022] The main process determines the optimal number of concurrent transmissions and creates and starts a number of transmission subprocesses that match the optimal number of concurrent transmissions.

[0023] Each transmission subprocess uses a dynamic pull mode to continuously request transmission tasks in the pending state from the main process and executes file transfer, transferring each file in the file group corresponding to the transmission task from the source storage path to the target storage path, and repeats the request and execution until all transmission tasks in the persistent state database are processed.

[0024] Each transmission subprocess reports its current transmission task progress information to the main process in real time during the execution of the transmission task; the main process updates the status of the corresponding transmission task in the persistent state database based on the progress information reported by each transmission subprocess.

[0025] Optionally, the main process determines the optimal number of concurrent transmissions, specifically including:

[0026] The main process constructs a transmission time correlation model with the number of concurrent transmissions as the independent variable and the total transmission time as the dependent variable;

[0027] With the goal of minimizing the total transmission time, the correlation model of the total transmission time is subjected to extreme value truncation to obtain the optimal number of concurrent transmissions corresponding to the minimum total transmission time.

[0028] Another aspect of the present invention provides a parallel transmission device for radio astronomy data, comprising:

[0029] The traversal module is used to traverse the source directory through the main process to obtain the metadata of all files in the source directory and the total number of files; the metadata of each file includes the file's identifier, size and source storage path;

[0030] The grouping module is used by the main process to group the files in the source directory according to the size and total number of files, resulting in multiple file groups, each file group containing at least one file;

[0031] The generation module is used to generate a transmission task for each file group through the main process, and write each transmission task to a disk queue file for persistent storage; each transmission task includes a task identifier, the identifier of each file in the file group corresponding to the transmission task, the source storage path, and the target storage path;

[0032] The scheduling module is used to schedule multiple parallel-running transmission subprocesses through the main process. Each transmission subprocess executes a transmission task within a disk queue file. Each transmission subprocess executes one transmission task at a time to transfer each file in the filegroup corresponding to the transmission task from the source storage path of the file to the target storage path.

[0033] Optionally, the main process groups the files according to their size and total number in the source directory, resulting in multiple file groups, specifically including:

[0034] The coefficient of variation is determined based on the size of each file in the source directory and the total number of files.

[0035] The correction factor is determined based on the coefficient of variation;

[0036] Files in the source directory whose size is greater than a preset size threshold are marked as oversized files, and all other files except the oversized files are marked as ordinary files. The number of oversized files in the source directory is counted.

[0037] The number of file groups is determined based on the correction factor, the preset size threshold, the total number of files in the source directory, and the number of oversized files in the source directory;

[0038] Each extremely large file is divided into a separate file group. All regular files are distributed to the remaining file groups after deducting the file groups corresponding to the extremely large files. During the distribution, the size of each regular file is used to balance the distribution so that the total amount of data in each remaining file group is similar, and the number of files in each file group does not exceed the upper limit of the number of files in a single group.

[0039] Optionally, the number of file groups satisfies the following relationship:

[0040] ,

[0041] in, It is the floor function. This is the maximum number of files in a single group. For the preset size threshold, This represents the number of extremely large files in the source directory. This represents the number of file groups. The correction factor is N, where N is the total number of files in the source directory and S is the total data volume of each file in the source directory.

[0042] Optionally, the main process schedules multiple parallel-running transmission subprocesses, each of which executes a transmission task within a disk queue file. Each transmission subprocess executes one transmission task at a time, transferring each file in the filegroup corresponding to that transmission task from its source storage path to its target storage path. Specifically, this includes:

[0043] The main process marks the initial state of all transmission tasks as the pending execution state and writes the state of all transmission tasks into the persistent state database.

[0044] The main process determines the optimal number of concurrent transmissions and creates and starts a number of transmission subprocesses that match the optimal number of concurrent transmissions.

[0045] Each transmission subprocess uses a dynamic pull mode to continuously request transmission tasks in the pending state from the main process and executes file transfer, transferring each file in the file group corresponding to the transmission task from the source storage path to the target storage path, and repeats the request and execution until all transmission tasks in the persistent state database are processed.

[0046] Each transmission subprocess reports its current transmission task progress information to the main process in real time during the execution of the transmission task; the main process updates the status of the corresponding transmission task in the persistent state database based on the progress information reported by each transmission subprocess.

[0047] Optionally, the main process determines the optimal number of concurrent transmissions, specifically including:

[0048] The main process constructs a transmission time correlation model with the number of concurrent transmissions as the independent variable and the total transmission time as the dependent variable;

[0049] With the goal of minimizing the total transmission time, the correlation model of the total transmission time is subjected to extreme value truncation to obtain the optimal number of concurrent transmissions corresponding to the minimum total transmission time. Attached Figure Description

[0050] Figure 1 A flowchart of a method for parallel transmission of radio astronomy data according to an embodiment of the present invention;

[0051] Figure 2 This is a structural block diagram of a radio astronomy data transmission device according to an embodiment of the present invention. Detailed Implementation

[0052] The preferred embodiments of the present invention are given below with reference to the accompanying drawings and described in detail.

[0053] Radio astronomy data comprises raw and derived astronomical observation data acquired by various radio telescopes through spatial observation, spectrum acquisition, time-series sampling, and interferometry. It encompasses baseband sampling data, visibility measurement data, spectral data, time-series pulse data, imaging data, and accompanying observation logs, calibration parameters, and other data types. This type of data is typically characterized by its large volume, highly heterogeneous single-file sizes, and massive number of files. It serves as core foundational data for astronomical research such as celestial evolution, deep-space transient source detection, galaxy surveys, and pulsar timing. In the radio astronomy observation system, all field observation equipment is deployed at remote field stations (transmitting ends). Due to the limitations of the station deployment environment, local storage capacity, computing resources, and network bandwidth are all limited, making it impossible to sustain the storage and refined scientific processing of massive amounts of observation data for extended periods. Therefore, all radio astronomy observation data needs to be reliably, efficiently, and completely transmitted from field observation stations to a high-performance data center (receiving end) to complete data archiving, unified calibration, joint imaging, scientific data mining, and long-term storage.

[0054] like Figure 1 As shown, this embodiment of the invention provides a method for parallel transmission of radio astronomy data, which is applied at the transmitting end and includes the following steps:

[0055] S100: The main process traverses the source directory to obtain the metadata of all files in the source directory and the total number of files; the metadata of each file includes the file's identifier, size and source storage path.

[0056] The source directory is a dedicated storage directory on the sending end used to store raw observation data and processed derived data. This directory can contain single-level files or multi-level nested subdirectories. The directory contains several data files to be transmitted across nodes. Storage media is not limited; it can be any readable and writable storage medium such as local hard drives, solid-state drives, disk arrays, or distributed storage mount directories. Each data file within the directory is an independent data unit related to radio astronomy operations. File types are not limited, including but not limited to raw observation sampling files, calibration data files, imaging processing result files, metadata record files, and spectrum analysis files. Each file has an independent and complete file entity, and different files can have different file sizes, ranging from KB-level small index files to MB-level intermediate processing files, and even GB-level or TB-level ultra-large raw observation files. All files are the objects to be transmitted in this cross-site transmission.

[0057] By traversing the source directory, the metadata of each file within it can be obtained, and the total number of files contained in the source directory can be counted. The metadata of each file includes at least the file's identifier, size, and source storage path.

[0058] S200: The main process groups the files in the source directory according to their size and total number, resulting in multiple file groups, each containing at least one file.

[0059] In some embodiments, the files are grouped according to their size and total number in the source directory to obtain multiple file groups, specifically including:

[0060] S210: Determine the coefficient of variation based on the size of each file in the source directory and the total number of files.

[0061] Assume the set of files in the source directory is ,in For the total number of files, each file Size The total data size of all files in the source directory is Based on this, the following statistics can be calculated:

[0062] (1) Average file size: (2) Standard deviation of file size: (3) Coefficient of variation: (Used to measure the relative dispersion of file sizes, i.e., heterogeneity); (4) Ratio of largest file size to mean: (Used to measure the dominance of a single file).

[0063] S220: Determine the correction factor based on the coefficient of variation.

[0064] Assuming the correction factor is It is related to the coefficient of variation and is used to adjust the number of filegroups to ensure that the fineness of filegroup partitioning matches the degree of heterogeneity. Its determination method is as follows:

[0065]

[0066] in, Basic correction coefficient, For the maximum correction factor, The growth slope coefficient, The lower limit of the low heterogeneity threshold, These parameters, which are lower limits for high heterogeneity thresholds, can be determined empirically before actual transmission, after extracting key distribution features through lightweight statistical scanning of the radio astronomy data to be transmitted.

[0067] S230: Mark files in the source directory whose size is greater than a preset size threshold as oversized files, mark the remaining files other than oversized files as normal files, and count the number of oversized files in the source directory.

[0068] Specifically, each file in the source directory can be traversed, and its size can be compared with a preset size threshold. If the file size is greater than the preset size threshold, the file is marked as an oversized file; otherwise, the file is marked as a normal file.

[0069] S240: Determine the number of file groups based on the correction factor, preset size threshold, total number of files in the source directory, and number of oversized files in the source directory.

[0070] The method for calculating the number of file groups is as follows:

[0071]

[0072] in, It is the floor function. This is the maximum number of files in a single group (which can be determined based on actual transmission conditions, such as available memory, number of CPU cores, file system characteristics, etc.). For the preset size threshold, This represents the number of extremely large files in the source directory. This represents the number of file groups.

[0073] S250: Divide each very large file into a separate file group, and allocate all ordinary files to the remaining file groups after deducting the file groups corresponding to each very large file. During allocation, the allocation is balanced based on the size of each ordinary file so that the total amount of data in each remaining file group tends to be the same, and the number of files in each file group does not exceed the upper limit of the number of files in a single group.

[0074] In step S250, the ordinary files can be sorted in descending order of file size, and then a greedy binning algorithm can be used to distribute them to the remaining files. Within a filegroup, the following conditions must be met during allocation: The total amount of data in each filegroup tends to be the same, and the number of files in each filegroup does not exceed the upper limit of the number of files in a single filegroup.

[0075] S300: The main process generates a transfer task for each file group and writes each transfer task to a disk queue file for persistent storage; each transfer task includes a task identifier, the identifier of each file in the file group corresponding to the transfer task, the source storage path, and the destination storage path.

[0076] After grouping, the main process can generate a corresponding transfer task for each file group. For example, if there are m file groups, there will be m transfer tasks. Each transfer task includes a task identifier, the identifier of each file in the file group corresponding to the transfer task, the source storage path, and the destination storage path. Each time a transfer task is generated, the main process writes it to a disk queue file for persistent storage. This way, when the transfer is interrupted and restarted, the main process can directly read the disk queue to restore all tasks without recursively traversing the radio astronomy source directory containing millions of files, repeatedly extracting file metadata, and regrouping. This greatly saves the time spent on task reconstruction in scenarios with massive numbers of files and avoids the computational waste caused by repeated calculations.

[0077] S400: The main process schedules multiple parallel-running transfer subprocesses, each of which executes transfer tasks within the disk queue file. Each transfer subprocess executes one transfer task at a time to transfer each file in the filegroup corresponding to the transfer task from the source storage path to the target storage path.

[0078] In some embodiments, step S400 specifically includes:

[0079] S410: The main process marks the initial state of all transmission tasks as pending execution and writes the state of all transmission tasks into the persistent state database.

[0080] S420: The main process determines the optimal number of concurrent transmissions, creates and starts a number of transmission child processes that match the optimal number of concurrent transmissions;

[0081] S430: Each transmission subprocess uses a dynamic pull mode to continuously request transmission tasks in the pending state from the main process and executes file transfer, transferring each file in the file group corresponding to the transmission task from the source storage path to the target storage path, and repeats the request and execution until all transmission tasks in the persistent state database are processed.

[0082] S440: Each transmission subprocess reports the progress information of the current transmission task to the main process in real time during the execution of the transmission task; the main process updates the status of the corresponding transmission task in the persistent state database in a unified manner based on the progress information reported by each transmission subprocess.

[0083] In step S420, the main process calculates the optimal number of concurrent transmissions by combining the system's available computing power, network bandwidth, storage read / write performance, and the total amount of data in the files to be transmitted. This value can be adaptively adjusted according to the business scenario, without a fixed upper or lower limit. Based on this number of concurrent transmissions, the main process creates and starts a corresponding number of resident transmission child processes in batches. After all child processes are started, they automatically enter a loop standby logic and remain resident throughout the process without being automatically destroyed. Compared to a static configuration scheme with a fixed number of child processes, adaptively determining the number of concurrent transmissions can avoid excessive concurrent processes competing for system resources and can also prevent insufficient concurrent transmissions from causing idle transmission bandwidth. It also caters to both low-computing-power field observation sites and high-performance data centers, making it more widely applicable.

[0084] The optimal number of concurrent transmissions is determined as follows:

[0085] Set a task The transmission time is (Obtained by dividing the total data volume of the task by the estimated network bandwidth), the number of concurrent transmissions is... Task scheduling adopts a dynamic pull mode, and under ideal conditions, the load of each working process is balanced, with a total completion time of:

[0086]

[0087] In the formula, The total transmission time, The time consumed by all non-data transfer operations (such as process creation and destruction, task queue access, etc.) can be estimated empirically. This is the scheduling efficiency factor, representing the percentage of time the actual scheduler achieves ideal parallel efficiency. It can be determined using empirical values. This is the load imbalance factor defined in this invention.

[0088] The expression is as follows:

[0089]

[0090] In the formula These are empirical calibration coefficients; The scheduling overhead caused by the increase in the number of files; This is due to load imbalance caused by variations in file size. The upper limit of parallelism when a single extremely large file dominates the total data volume, when When the value is close to 1, meaning a single file is absolutely dominant, It increases significantly, and the parallel speedup ratio approaches 1.

[0091] Finding the minimum value of p represents the optimal number of concurrent transmissions. For example, this can be determined based on... The calculation formula is to differentiate p with respect to p, and then set the derivative to 0 to obtain the value of p, which is the optimal transmission concurrency. For example, the optimal transmission concurrency is 8.

[0092] Load imbalance factor It can quantify the relationship between the degree of data heterogeneity and the parallel speedup ratio, and is used for scheduling parameter optimization and performance prediction.

[0093] In step S430, a dynamic pull scheduling mechanism is adopted, where child processes actively request tasks. After all resident transmission child processes finish their current tasks, they continuously send task request requests to the main process. Upon receiving a request, the main process filters transmission tasks with a pending status and distributes them to the corresponding child processes. After a child process receives a transmission task, it sequentially traverses all files in the file group corresponding to that task, reads the radio astronomy data files under the file source storage path, and transmits them to the target storage path at the receiving end through a preset transmission channel. After completing a set of tasks, the child process immediately initiates another request, and this process repeats. If the number of remaining pending tasks is less than the total number of currently running child processes, child processes with no tasks to receive will remain idle and continuously send request requests until there are no transmission tasks with a pending status in the persistent state database. This dynamic pull mode can automatically balance the load of each child process. For scenarios involving the transmission of TB-level ultra-large observation files, which take longer, idle child processes with faster processing speeds can continuously take on the remaining tasks, preventing the problem of some processes being under high load for a long time while others are idle and wasting resources.

[0094] In step S440, each transmission subprocess continuously collects real-time progress information throughout the entire file transmission process. The reported content includes, but is not limited to, the current task ID, the number of files transmitted, the percentage of single file transmission completed, the total amount of data transmitted, details of failed files, and transmission time, among other multi-dimensional progress data. Reporting methods can be either periodically scheduled or triggered by single file transmission completion, allowing for flexible configuration. All status update operations are uniformly executed by the main process. The transmission subprocesses are only responsible for progress reporting and do not directly modify the persistent state database data. This effectively avoids data corruption and state overwriting issues caused by concurrent read / write operations by multiple subprocesses to persistent storage, ensuring the consistency of task data within the persistent state database. Maintenance personnel can view the global transmission progress and locate failed tasks in real time by reading the persistent state database, facilitating manual intervention and retrying of abnormal transmission tasks. This is suitable for the 24 / 7 uninterrupted data transmission maintenance scenario of the observatory.

[0095] like Figure 2 As shown, this embodiment of the invention also provides a parallel transmission device for radio astronomy data, which includes a traversal module 10, a grouping module 20, a generation module 30, and a scheduling module 40.

[0096] The traversal module 10 is used to traverse the source directory through the main process to obtain the metadata of all files in the source directory and the total number of files; wherein, the metadata of each file includes the file's identifier, size and source storage path.

[0097] The grouping module 20 is used by the main process to group the files according to the size and total number of files in the source directory to obtain multiple file groups, each file group including at least one file.

[0098] The generation module 30 is used to generate a transmission task for each file group through the main process and write each transmission task to a disk queue file for persistent storage; each transmission task includes a task identifier, the identifier of each file in the file group corresponding to the transmission task, the source storage path and the target storage path.

[0099] The scheduling module 40 is used to schedule multiple parallel-running transmission subprocesses through the main process. Each transmission subprocess executes a transmission task in the disk queue file. Each transmission subprocess executes one transmission task at a time to transfer each file in the file group corresponding to the transmission task from the source storage path of the file to the target storage path.

[0100] The traversal module 10, grouping module 20, generation module 30 and scheduling module 40 are functional modules corresponding to steps S100-S400 in the method embodiment, respectively. Their specific implementation methods can be found in the description in the method embodiment, and will not be repeated here.

[0101] The method and apparatus for parallel transmission of radio astronomy data according to embodiments of the present invention adaptively calculates correction factors based on statistical characteristics of file size distribution through a heterogeneous sensing adaptive task partitioning strategy. The system dynamically adjusts the number of task blocks, isolating extremely large files into independent single-file blocks to achieve load balancing under highly heterogeneous data distribution. This solves the problems of uneven load and performance degradation in parallel transmission under extremely heterogeneous file distribution, enabling high-throughput data transmission. A lightweight task scheduling mechanism (using zero-copy buffer management and segmented metadata caching) eliminates the rolling checksum and computational overhead of traditional rsync, keeping CPU utilization at a low level and avoiding resource competition with scientific computing tasks at the station. By persisting each transmission task to disk, the overhead of re-traversing the directory after transmission interruption is eliminated, achieving second-level task recovery. An extremely low memory footprint design (using a task-level memory management strategy, storing the task queue on disk, loading the current task's file list into memory during runtime, and releasing it after processing; simultaneously employing kernel-level zero-copy data forwarding to avoid allocating user-mode cache based on file size) ensures stable operation under limited hardware resources at field stations.

[0102] Another embodiment of the present invention provides a readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the steps of the radio astronomy data parallel transmission method in the above embodiments of the present invention.

[0103] Another embodiment of the present invention provides an electronic device, which includes a memory and a processor. The memory stores executable code. When the processor executes the executable code, it performs the steps of the radio astronomy data parallel transmission method in the above embodiments of the present invention.

[0104] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0105] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this invention, the functions of each unit can be implemented in one or more software and / or hardware components.

[0106] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0112] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by electronic devices. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0116] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Various variations can be made to the above embodiments of the present invention. That is, all simple and equivalent changes and modifications made based on the claims and description of this invention fall within the protection scope of the claims of this patent. All aspects not described in detail in this invention are conventional technical content.

Claims

1. A method for parallel transmission of radio astronomy data, characterized in that, include: The main process traverses the source directory to obtain the metadata of all files in the source directory and the total number of files; the metadata of each file includes the file's identifier, size, and source storage path; The main process groups the files according to the size and total number of files in the source directory to obtain multiple file groups, each file group containing at least one file; The main process generates a transfer task for each file group and writes each transfer task to a disk queue file for persistent storage; each transfer task includes a task identifier, the identifier of each file in the file group corresponding to the transfer task, the source storage path, and the target storage path; The main process schedules multiple parallel-running transmission subprocesses, each of which executes a transmission task within a disk queue file. Each transmission subprocess executes one transmission task at a time to transfer each file in the filegroup corresponding to the transmission task from the source storage path to the target storage path.

2. The method for parallel transmission of radio astronomy data according to claim 1, characterized in that, The main process groups the files in the source directory according to their size and total number, resulting in multiple file groups, specifically including: The coefficient of variation is determined based on the size of each file in the source directory and the total number of files. The correction factor is determined based on the coefficient of variation; Files in the source directory whose size is greater than a preset size threshold are marked as oversized files, and all other files except the oversized files are marked as ordinary files. The number of oversized files in the source directory is counted. The number of file groups is determined based on the correction factor, the preset size threshold, the total number of files in the source directory, and the number of oversized files in the source directory; Each extremely large file is divided into a separate file group. All regular files are distributed to the remaining file groups after deducting the file groups corresponding to the extremely large files. During the distribution, the size of each regular file is used to balance the distribution so that the total amount of data in each remaining file group is similar, and the number of files in each file group does not exceed the upper limit of the number of files in a single group.

3. The method for parallel transmission of radio astronomy data according to claim 2, characterized in that, The number of file groups satisfies the following relationship: ， in, It is the floor function. This is the maximum number of files in a single group. For the preset size threshold, This represents the number of extremely large files in the source directory. This represents the number of file groups. The correction factor is N, where N is the total number of files in the source directory and S is the total data volume of each file in the source directory.

4. The method for parallel transmission of radio astronomy data according to claim 1, characterized in that, The main process schedules multiple parallel-running transfer subprocesses, each of which executes transfer tasks within the disk queue file. Each transfer subprocess executes one transfer task at a time, transferring each file in the filegroup corresponding to that transfer task from its source storage path to its target storage path. Specifically, this includes: The main process marks the initial state of all transmission tasks as the pending execution state and writes the state of all transmission tasks into the persistent state database. The main process determines the optimal number of concurrent transmissions and creates and starts a number of transmission subprocesses that match the optimal number of concurrent transmissions. Each transmission subprocess uses a dynamic pull mode to continuously request transmission tasks in the pending state from the main process and executes file transfer, transferring each file in the file group corresponding to the transmission task from the source storage path to the target storage path, and repeats the request and execution until all transmission tasks in the persistent state database are processed. Each transmission subprocess reports its current transmission task progress information to the main process in real time during the execution of the transmission task; the main process updates the status of the corresponding transmission task in the persistent state database based on the progress information reported by each transmission subprocess.

5. The method for parallel transmission of radio astronomy data according to claim 4, characterized in that, The main process determines the optimal number of concurrent transmissions, specifically including: The main process constructs a transmission time correlation model with the number of concurrent transmissions as the independent variable and the total transmission time as the dependent variable; With the goal of minimizing the total transmission time, the correlation model of the total transmission time is subjected to extreme value truncation to obtain the optimal number of concurrent transmissions corresponding to the minimum total transmission time.

6. A parallel transmission device for radio astronomy data, characterized in that, include: The traversal module is used to traverse the source directory through the main process to obtain the metadata of all files in the source directory and the total number of files; the metadata of each file includes the file's identifier, size and source storage path; The grouping module is used by the main process to group the files in the source directory according to the size and total number of files, resulting in multiple file groups, each file group containing at least one file; The generation module is used to generate a transmission task for each file group through the main process, and write each transmission task to a disk queue file for persistent storage; each transmission task includes a task identifier, the identifier of each file in the file group corresponding to the transmission task, the source storage path, and the target storage path; The scheduling module is used to schedule multiple parallel-running transmission subprocesses through the main process. Each transmission subprocess executes a transmission task within a disk queue file. Each transmission subprocess executes one transmission task at a time to transfer each file in the filegroup corresponding to the transmission task from the source storage path of the file to the target storage path.

7. The parallel transmission device for radio astronomy data according to claim 6, characterized in that, The main process groups the files in the source directory according to their size and total number, resulting in multiple file groups, specifically including: The coefficient of variation is determined based on the size of each file in the source directory and the total number of files. The correction factor is determined based on the coefficient of variation; Files in the source directory whose size is greater than a preset size threshold are marked as oversized files, and all other files except the oversized files are marked as ordinary files. The number of oversized files in the source directory is counted. The number of file groups is determined based on the correction factor, the preset size threshold, the total number of files in the source directory, and the number of oversized files in the source directory; Each extremely large file is divided into a separate file group. All regular files are distributed to the remaining file groups after deducting the file groups corresponding to the extremely large files. During the distribution, the size of each regular file is used to balance the distribution so that the total amount of data in each remaining file group is similar, and the number of files in each file group does not exceed the upper limit of the number of files in a single group.

8. The parallel transmission device for radio astronomy data according to claim 7, characterized in that, The number of file groups satisfies the following relationship: ， in, It is the floor function. This is the maximum number of files in a single group. For the preset size threshold, This represents the number of extremely large files in the source directory. This represents the number of file groups. The correction factor is N, where N is the total number of files in the source directory and S is the total data volume of each file in the source directory.

9. The parallel transmission device for radio astronomy data according to claim 6, characterized in that, The main process schedules multiple parallel-running transfer subprocesses, each of which executes transfer tasks within the disk queue file. Each transfer subprocess executes one transfer task at a time, transferring each file in the filegroup corresponding to that transfer task from its source storage path to its target storage path. Specifically, this includes: The main process marks the initial state of all transmission tasks as the pending execution state and writes the state of all transmission tasks into the persistent state database. The main process determines the optimal number of concurrent transmissions and creates and starts a number of transmission subprocesses that match the optimal number of concurrent transmissions. Each transmission subprocess uses a dynamic pull mode to continuously request transmission tasks in the pending state from the main process and executes file transfer, transferring each file in the file group corresponding to the transmission task from the source storage path to the target storage path, and repeats the request and execution until all transmission tasks in the persistent state database are processed. Each transmission subprocess reports its current transmission task progress information to the main process in real time during the execution of the transmission task; the main process updates the status of the corresponding transmission task in the persistent state database based on the progress information reported by each transmission subprocess.

10. The parallel transmission device for radio astronomy data according to claim 9, characterized in that, The main process determines the optimal number of concurrent transmissions, specifically including: The main process constructs a transmission time correlation model with the number of concurrent transmissions as the independent variable and the total transmission time as the dependent variable; With the goal of minimizing the total transmission time, the correlation model of the total transmission time is subjected to extreme value truncation to obtain the optimal number of concurrent transmissions corresponding to the minimum total transmission time.

Images