Method and System for Processing Computational Tasks under the HTCondor Framework

Abstract

The present disclosure provides a method and system for processing computing tasks under the HTCondor framework, which belongs to the field of supercomputing technology. The method includes: obtaining a task to be processed and the corresponding data file directory structure; extracting the directory structure of the data file and encoding it; renaming the data file based on the encoded directory structure; distributing the task to be processed and the renamed data file to a computing node based on the task distribution mechanism of HTCondor; the computing node decodes the data file name, creates a complete directory structure, and performs the calculation of the task to be processed, generating a calculation result file in the corresponding directory; encoding the directory structure of the calculation result file and naming it based on the encoded directory structure; based on the return mechanism of HTCondor, returning the obtained calculation result file, and storing the calculation result in the corresponding directory based on the decoded directory structure.

Description

Technical Field

[0001] The present disclosure belongs to the field of supercomputing technology, and particularly relates to a method and system for processing computing tasks under the HTCondor framework. Background Art

[0002] The statements in this part merely provide background technical information related to the present disclosure and do not necessarily constitute prior art.

[0003] The design purpose of HTCondor is to integrate computing resources. It can build computing resources including schools, supercomputing centers, and other organizations into an HTCondor pool, and each resource will be used as a sub-node to be called by the platform. In this way, the computing platform has the ability to cross-cluster call computing resources; when the platform issues a large-scale intensive computing task, HTCondor will split it into individual sub-tasks. And select appropriate computing resources in the HTCondor pool to calculate appropriate computing tasks.

[0004] The inventors found that due to the preciousness of bandwidth resources in Internet data transmission, HTCondor provides fine-grained transmission control based on this, and users need to manually specify the transmitted data; however, under normal circumstances, the transmitted data specified by users is usually stored in different folders. In order to use precious bandwidth and IO resources to transmit useful data files as much as possible, the original directory structure of the data files will be ignored, resulting in the inability of the pre-developed HTCondor script to read the correct data. That is, in the existing processing process during job distribution and data transmission back to the computing node, the transmitted data is data files without directory structure, which is suitable for simple calculations with fewer data files and a single directory structure, and cannot meet the job computing tasks with complex directory structures, bringing great inconvenience to users. Summary of the Invention

[0005] In order to solve the above problems, the present disclosure provides a method and system for processing computing tasks under the HTCondor framework. The solution encodes the file name based on the sub-path of the data file, effectively solves the problem that the existing HTCondor framework is not applicable in the case of complex data file directory structures, and facilitates the use of users.

[0006] According to the first aspect of the embodiments of the present disclosure, a method for processing computing tasks under the HTCondor framework is provided. The method is applied to a computing system composed of several computing nodes installed with the HTCondor framework. The method includes:

[0007] Obtain the task to be processed and the corresponding data file directory structure;

[0008] Extract the directory structure of the data file and encode it; rename the data file based on the encoded directory structure;

[0009] Based on the task distribution mechanism of HTCondor, distribute the tasks to be processed and the renamed data files to the computing nodes;

[0010] The computing node decodes the data file name, creates a complete directory structure, and performs the calculation of the tasks to be processed, generating a calculation result file in the corresponding directory;

[0011] Encode the directory structure of the calculation result file and name it based on the encoded directory structure;

[0012] Based on the return mechanism of HTCondor, return the obtained calculation result file, and store the calculation result file in the corresponding directory based on the decoded directory structure.

[0013] Further, the extraction of the directory structure of the data file and the encoding are specifically as follows: recursively traverse the directory structure of the data file required for the calculation to obtain a complete directory structure; implement the encoding of the directory structure by concatenating each level of the directory.

[0014] Further, the computing node decodes the data file name and creates a complete directory structure. The decoding is specifically as follows: based on the pre-set encoding rule, perform an inverse operation on the data file name to decode it and obtain a complete directory structure.

[0015] Further, the concatenation of each level of the directory connects each level of the directory through a separator, and the separator is an underscore or a hyphen.

[0016] Further, the tasks to be processed are provided by the user in the form of a job script file.

[0017] According to the second aspect of the embodiments of the present disclosure, a computing task processing system based on the HTCondor framework is provided, which is applied to a computing system composed of several computing nodes installed with the HTCondor framework. The system includes:

[0018] A data acquisition unit, which is used to acquire the tasks to be processed and the corresponding data file directory structure;

[0019] An encoding unit, which is used to extract the directory structure of the data file and encode it; rename the data file based on the encoded directory structure;

[0020] A task processing unit, which is used to distribute the tasks to be processed and the renamed data files to computing nodes based on the task distribution mechanism of HTCondor; the computing nodes decode the data file names, create a complete directory structure, and perform the calculation of the tasks to be processed, generating calculation result files in the corresponding directories;

[0021] A calculation result feedback unit, which is used to encode the directory structure of the calculation result files and name them based on the encoded directory structure; based on the feedback mechanism of HTCondor, it feedbacks the obtained calculation result files, and stores the calculation result files in the corresponding directories based on the decoded directory structure.

[0022] According to the third aspect of the embodiments of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program running on the memory. When the processor executes the program, it implements the described calculation task processing method based on the HTCondor framework.

[0023] According to the fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it implements the described calculation task processing method based on the HTCondor framework.

[0024] Compared with the prior art, the beneficial effects of the present disclosure are:

[0025] (1) The solution of the present disclosure provides a calculation task processing method and system for the HTCondor framework. The solution encodes the file names based on the sub-paths of the data files, effectively solving the problem that the existing HTCondor framework is not applicable when the data file directory structure is complex, and facilitating the use of users;

[0026] (2) The solution of the present disclosure can process jobs with complex data directories without user perception, and by providing the method of encoding file names with sub-paths, it can reduce the pressure on bandwidth and IO on the premise of ensuring the complete transmission of the data file structure;

[0027] (3) In the process of writing the HTCondor program, the solution of the present disclosure does not need to specifically process the steps for reading data files, reducing the difficulty of use for users and improving the readability of the code.

[0028] The advantages of the additional aspects of the present disclosure will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of the present disclosure. Description of the Drawings

[0029] The accompanying drawings forming a part of this disclosure are used to provide a further understanding of this disclosure. The schematic embodiments and descriptions thereof of this disclosure are used to explain this disclosure and do not constitute an improper limitation to this disclosure.

[0030] Figure 1 It is a flowchart of a method for processing computing tasks in the prior art described in the embodiments of this disclosure based on the HTCondor framework;

[0031] Figure 2 It is a flowchart of an improved method for processing computing tasks based on the HTCondor framework described in the embodiments of this disclosure. Detailed implementation manners

[0032] The following further describes this disclosure in conjunction with the accompanying drawings and embodiments.

[0033] It should be noted that the following detailed descriptions are all illustrative and are intended to provide a further description of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this disclosure belongs.

[0034] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to this disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0035] Without conflict, the embodiments in this disclosure and the features in the embodiments can be combined with each other.

[0036] Embodiment 1:

[0037] The purpose of this embodiment is to provide a method for processing computing tasks under the HTCondor framework.

[0038] A method for processing computing tasks under the HTCondor framework, which is applied to a computing system composed of several computing nodes installed with the HTCondor framework. The method includes:

[0039] Obtain the task to be processed and the corresponding data file directory structure;

[0040] Extract the directory structure of the data file and encode it; Rename the data file based on the encoded directory structure;

[0041] Based on the task distribution mechanism of HTCondor, distribute the task to be processed and the renamed data file to the computing nodes;

[0042] The computing node decodes the data file name, creates a complete directory structure, and performs the calculation of the task to be processed, generating a calculation result file in the corresponding directory.

[0043] Encode the directory structure of the calculation result file and name it based on the encoded directory structure.

[0044] Based on the return mechanism of HTCondor, the obtained calculation result file is returned, and based on the decoded directory structure, the calculation result file is stored in the corresponding directory.

[0045] Further, extracting the directory structure of the data file and encoding it specifically includes: recursively traversing the directory structure of the data file required for calculation to obtain a complete directory structure; realizing the encoding of the directory structure by concatenating directories at all levels.

[0046] Further, the computing node decodes the data file name and creates a complete directory structure. The decoding specifically is: based on the preset encoding rule, through the inverse operation, decode the data file name to obtain a complete directory structure.

[0047] Further, the concatenation of directories at all levels connects the directories at all levels through a separator, and the separator is an underscore or a hyphen.

[0048] Further, the task to be processed is provided by the user in the form of a job script file.

[0049] Specifically, for the sake of understanding, the following combines the accompanying drawings to elaborate on the solution described in this disclosure in detail:

[0050] First, as Figure 1 shown, it shows the job submission and calculation process of the existing general HTCondor. Its technical concept is: the user provides a job script, an input file, and its file directory structure, and then submits the job to the target queue through the HTCondor program; the main node (job scheduling node) of HTCondor distributes the calculation task to the HTCondor sub-node (computing node); after the sub-node performs the calculation, it returns the calculation result to the main node. The specific processing process includes the following steps:

[0051] Step (1): The user provides a job script, an input file, and a file directory structure.

[0052] Step (2): The user calls the HTCondor program to submit a job.

[0053] Step (3): The HTCondor master node (i.e., the user node, which is used for the user to issue job tasks and data files) distributes jobs to the child nodes (computing nodes, and the computing nodes can be general-purpose PCs or servers), and transmits input files without an attached directory structure.

[0054] Step (4): The HTCondor child node calls the computing software required for the computing task (such as the computing software related to VASP (Vienna Ab-inito Simulation Package)), and performs the calculation.

[0055] Step (5): After the calculation is completed, the HTCondor child node transmits the calculation results back to the master node.

[0056] Step (6): The HTCondor master node integrates the data transmitted back by each child node.

[0057] Step (7): The job is completed.

[0058] In the above existing method, during the job distribution of the master node and the transmission of the calculation data of the child node, the transmitted data are all data files without a directory structure. Therefore, this method is only applicable to simple calculations with fewer data files and a single directory structure, and cannot meet the job calculation tasks with complex directory structures. To solve the defect that HTCondor cannot transmit the directory structure of data files, this embodiment proposes a scheme based on sub-path encoded file names. Specifically, as Figure 2 shown, a method for processing computing tasks under the HTCondor framework specifically includes the following steps:

[0059] Step (1): The user provides a job script and input files: / root / A / b.data, / root / C / d.data, and calls this method.

[0060] Step (2): Use the recursive search method to extract the file directory structure of the files provided by the user, and add decoding and encoding execution commands at the beginning and end of the user job respectively; wherein, the encoding is specifically: realizing the encoding of the directory structure by concatenating each level of directories; the concatenation of each level of directories is to connect each level of directories through a connection symbol, and the connection symbol is an underscore or a hyphen; the decoding is specifically: based on a pre-set encoding rule, through an inverse operation, decoding the data file name to obtain the complete directory structure.

[0061] Step (3): Encrypt the extracted file directory structure and rename the data files: / root / A / b.data >> root-A-b.data, / root / C / d.data >> root-C-d.data.

[0062] Step (4): Call the HTCondor program to submit jobs.

[0063] Step (5): The HTCondor master node distributes jobs to child nodes and transfers the renamed data files.

[0064] Step (6): After receiving the data files, the child nodes execute the job script, decode the data file names of the child nodes, and create a complete directory structure: root-A-b.data >> / root / A / b.data, root-C-d.data >> / root / C / d.data.

[0065] Step (7): Continue to execute the job script, call the computing software required for the computing task (such as VASP-related computing software), and perform the calculation.

[0066] Step (8): After the calculation is completed, encode the result files generated by the child nodes and send them back to the master node.

[0067] Step (9): The HTCondor master node decodes the data sent back by each child node, creates the correct directory structure, and stores the result files.

[0068] Step (10) The job is completed.

[0069] Embodiment 2:

[0070] The purpose of this embodiment is to provide a computing task processing system based on the HTCondor framework.

[0071] A computing task processing system based on the HTCondor framework, which is applied to a computing system composed of several computing nodes installed with the HTCondor framework. The system includes:

[0072] A data acquisition unit, which is used to acquire the task to be processed and the corresponding data file directory structure.

[0073] An encoding unit, which is used to extract the directory structure of the data files and perform encoding; rename the data files based on the encoded directory structure.

[0074] A task processing unit, which is used to distribute the task to be processed and the renamed data files to the computing nodes based on the task distribution mechanism of HTCondor; the computing nodes decode the data file names, create a complete directory structure, and perform the calculation of the task to be processed, generating calculation result files in the corresponding directory.

[0075] A calculation result feedback unit, which is used to encode the directory structure of the calculation result file and name it based on the encoded directory structure; based on the feedback mechanism of HTCondor, the obtained calculation result file is fed back, and based on the decoded directory structure, the calculation result file is stored in the corresponding directory.

[0076] Further, extracting the directory structure of the data file and encoding it specifically includes: recursively traversing the directory structure of the data file required for calculation to obtain the complete directory structure; encoding the directory structure by splicing each level of directory.

[0077] Further, the calculation node decodes the data file name to create a complete directory structure. The decoding specifically includes: based on the preset encoding rule, through reverse operation, decoding the data file name to obtain the complete directory structure.

[0078] Further, the system in this embodiment corresponds to the method in Embodiment 1. The specific technical details have been described in detail in Embodiment 1, so they will not be elaborated here.

[0079] In more embodiments, there is also provided:

[0080] An electronic device includes a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the method described in Embodiment 1 is completed. For the sake of brevity, it will not be elaborated here.

[0081] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0082] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0083] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the method described in Embodiment 1 is completed.

[0084] The method in the first embodiment can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0085] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with this embodiment can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this disclosure.

[0086] The above-described method and system for processing computing tasks under the HTCondor framework provided by the above embodiment can be implemented and have broad application prospects.

[0087] The foregoing is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, various changes and modifications can be made to the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for processing computing tasks under the HTCondor framework, which is applied to a computing system composed of several computing nodes installed with the HTCondor framework. Characterized in that, The method includes: Obtain the task to be processed and the corresponding data file directory structure; Extract the directory structure of the data file and encode it; Rename the data file based on the encoded directory structure; Based on the task distribution mechanism of HTCondor, distribute the task to be processed and the renamed data file to the computing nodes; The computing node decodes the data file name, creates a complete directory structure, and performs the calculation of the task to be processed, generating a calculation result file in the corresponding directory; Encode the directory structure of the calculation result file and name it based on the encoded directory structure; Based on the return mechanism of HTCondor, return the obtained calculation result file, and store the calculation result file in the corresponding directory based on the decoded directory structure.

2. A method for processing computing tasks under the HTCondor framework as described in claim 1, Characterized in that, The extraction of the directory structure of the data file and the encoding are specifically: recursively traverse the directory structure of the data file required for calculation to obtain a complete directory structure; realize the encoding of the directory structure by splicing each level of the directory.

3. A method for processing computing tasks under the HTCondor framework as described in claim 1, Characterized in that, The computing node decodes the data file name and creates a complete directory structure. The decoding is specifically: based on a pre-set encoding rule, perform an inverse operation on the data file name to decode it and obtain a complete directory structure.

4. A method for processing computing tasks under the HTCondor framework as described in claim 2, Characterized in that, The splicing of each level of the directory connects each level of the directory through a connection symbol, and the connection symbol is an underscore or a dash.

5. A method for processing computing tasks under the HTCondor framework as described in claim 1, Characterized in that, The task to be processed is provided by the user in the form of a job script file.

6. A computing task processing system based on the HTCondor framework, which is applied to a computing system composed of several computing nodes installed with the HTCondor framework. Characterized in that, The system includes: A data acquisition unit, which is used to acquire the task to be processed and the corresponding data file directory structure; An encoding unit, which is used to extract the directory structure of the data file and encode it; Rename the data file based on the encoded directory structure; A task processing unit, which is used to distribute the task to be processed and the renamed data file to the computing nodes based on the task distribution mechanism of HTCondor; The computing node decodes the data file name, creates a complete directory structure, and performs the calculation of the task to be processed, generating a calculation result file in the corresponding directory; A calculation result feedback unit, which is used to encode the directory structure of the calculation result file and name it based on the encoded directory structure; based on the feedback mechanism of HTCondor, the obtained calculation result file is fed back, and based on the decoded directory structure, the calculation result file is stored in the corresponding directory.

7. A calculation task processing system for the HTCondor framework as described in claim 6, characterized in that extracting the directory structure of the data file and encoding it specifically includes: recursively traversing the directory structure of the data file required for calculation to obtain the complete directory structure; realizing the encoding of the directory structure by concatenating each level of directory.

8. A calculation task processing system for the HTCondor framework as described in claim 6, characterized in that the calculation node decodes the data file name to create a complete directory structure, and the decoding specifically includes: based on the pre-set encoding rule, through reverse operation, decoding the data file name to obtain the complete directory structure.

9. An electronic device, including a memory, a processor, and a computer program running on the memory, characterized in that when the processor executes the program, it implements a calculation task processing method based on the HTCondor framework as described in any one of claims 1-5.

10. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that when the program is executed by the processor, it implements a calculation task processing method based on the HTCondor framework as described in any one of claims 1-5.

Images