Job scheduling system and method based on data parallelism

Abstract

The invention discloses a job scheduling system and method based on data parallelism. The job scheduling system provided by the specification comprises a task processing module and a high-efficiency calculation module, the task processing module is used for receiving a large-scale computing task delivered by a user, determining a task structure corresponding to the large-scale computing task, splitting the large-scale computing task according to the task structure and constructing a plurality of computing jobs; the calculation operation is sent to the high-efficiency calculation module; and the high-efficiency calculation module is used for receiving and processing the calculation operation sent by the task processing module, and the high-efficiency calculation module comprises a plurality of calculation cluster modules.

Description

Technical Field

[0001] The present specification relates to the field of computer technology, and in particular to a job scheduling system and method based on data parallelism. Background Art

[0002] Nowadays, the development and application of artificial intelligence and big data technologies have led to a surge in the amount of data. Usually, these data are processed by CPU-intensive algorithms or computing software. The computing model for processing these massive data is often to use a fixed computing process for each piece of data to obtain the result, that is, large-scale homogeneous computing.

[0003] The processing of a single computing task can be completed by one or more computing nodes, but in the face of massive computing tasks, computing clusters are needed to ensure the timeliness of the calculation. Each computing center usually contains one or more computing clusters, and each cluster has one or more computing cluster modules.

[0004] Currently, cluster or job management systems manage jobs for a single cluster, that is, they only manage jobs assigned to that cluster, and the clusters do not affect each other. This approach will lead to unbalanced cluster loads, and a single cluster processing too many tasks will reduce processing efficiency, seriously affecting computing timeliness.

[0005] Therefore, how to improve the utilization of computing resources and the timeliness of calculations is an urgent problem to be solved. Summary of the invention

[0006] This specification provides a job scheduling system and method based on data parallelism to at least partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions: This specification provides a job scheduling system, the system comprising: a task processing module and a high-performance computing module; The task processing module is used to receive the large-scale computing task delivered by the user, determine the task structure corresponding to the large-scale computing task, split the large-scale computing task according to the task structure and construct a plurality of computing jobs; and send the computing job to the high-performance computing module; The high-performance computing module is used to receive and process the computing jobs sent by the task processing module. The high-performance computing module includes several computing cluster modules.

[0008] Optionally, receiving large-scale computing tasks delivered by users, specifically including: Parsing configuration information of the large-scale computing task, the configuration information including the name of the large-scale computing task and several task data unit parameters, the task data unit parameters including but not limited to text input, file selection, and directory; The configuration information is stored, and the state of the large-scale computing task is set to a running state.

[0009] Optionally, determining a task structure corresponding to the large-scale computing task specifically includes: Determine a task structure according to the configuration information contained in the large-scale computing task, wherein the task structure is jointly described by a minimum computing task and a plurality of task data units, wherein the minimum computing task is used to characterize the execution process, input and output, and execution environment of the large-scale computing task, and the task data units include general task data units and key task data units, wherein the general task data units represent data that have no effect on the computing resources required for the actual computing task generated by adopting the task structure, and the key task data units represent data that have an effect on the computing resources required for the actual computing task generated by adopting the task structure; wherein the minimum computing task, the general task data units, and the key task data units are components for generating a computing job that can actually be run; Determine the unit information of each task data unit.

[0010] Optionally, determining unit information of each task data unit specifically includes: Determine the name, type, data value and data usage characteristics of each task data unit; The data value of the task data unit is decomposed according to the data usage characteristics to obtain task data entries, and the task data entries are marked as unprocessed, wherein the data value of the general task data unit is decomposed to obtain the general task data entry, and the data value of the critical task data unit is decomposed to obtain the critical task data entry.

[0011] Optionally, the large-scale computing task is split according to the task structure and a plurality of computing jobs are constructed, specifically including: determining the minimum computing resources required for the task structure; Obtaining available computing resources of the high-performance computing module; When the available computing resources are less than the minimum computing resources, skip the generation of this computing job; when the available computing resources are not less than the minimum computing resources, determine the number of key task data items that can be processed simultaneously by the high-performance computing module according to the minimum computing resources and the available computing resources; According to the number of the key task data entries that can be processed simultaneously, reading the key task data entries not exceeding the number as the key task data entries to be processed; parsing actual resource requirements of the computing job to be generated according to the unit information of the key task data unit in the task structure and the key task data entry to be processed; Reading parameter value information of the general task data unit according to the data usage characteristics of the general task data unit in the task structure; Generate a job startup file in a combined manner according to the minimum computing task, the key task data entry to be processed, the parameter value information of the general task data unit, and the actual resource demand; The computing job is constructed according to the job startup file.

[0012] Optionally, the task processing module is further used to periodically monitor the operating status of the high-performance computing module; In response to completion of any computing job executed by any high-performance computing module, marking the status of a key task item included in the computing job as completed; In response to an exception occurring in any high-performance computing module while processing a computing job, the task data entries included in the computing job in which the exception occurred are restored, the restored key task data entries are stored in a key task data unit data entry library, the general task data entries are stored in a general task data unit data entry library, and the status of the restored task data entries is marked as incomplete; In response to all the key task data items included in the large-scale computing task being in a completed state, marking the state of the large-scale computing task as completed.

[0013] Optionally, the high-performance computing module is further used to monitor resource information and hardware status of each computing cluster module included in the high-performance computing module.

[0014] Optionally, the system further comprises a data management module for accessing the data uploaded by the user, performing structured and / or unstructured storage on the data, and performing addition, deletion, rewriting and query operations on the data.

[0015] This specification provides a job scheduling method, which is applied to a task processing module in a job scheduling system, and includes: Receive large-scale computing tasks submitted by users; Determining a task structure corresponding to the large-scale computing task; Splitting the large-scale computing task according to the task structure and constructing a plurality of computing jobs; The computing operation is sent to a high performance computing module.

[0016] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned job scheduling method is implemented.

[0017] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned job scheduling method when executing the program.

[0018] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: The job scheduling system provided in this specification includes a task processing module and a high-performance computing module; the task processing module is used to receive large-scale computing tasks delivered by users, determine the task structure corresponding to the large-scale computing tasks, split the large-scale computing tasks according to the task structure and construct a number of computing jobs; send the computing jobs to the high-performance computing module; the high-performance computing module is used to receive and process the computing jobs sent by the task processing module, and the high-performance computing module includes a number of computing cluster modules.

[0019] The job scheduling system provided in this specification can split the received large-scale computing tasks through the task processing module, and construct the split task data entries into computing jobs that match the resources of the high-performance computing modules according to the resource information of each high-performance computing module, and the high-performance computing modules process the computing jobs. The system can be used to decompose complex large-scale tasks into task data entries and construct them into computing jobs that match the resources of the high-performance computing modules, so that the load brought by the large-scale computing tasks can be reasonably distributed to each high-performance computing module, effectively improving the utilization rate of computing resources and the processing efficiency of computing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings: Figure 1 A schematic diagram of the structure of a job scheduling system provided in this specification; Figure 2 A structural diagram of a task processing module provided in this specification; Figure 3 A flowchart of a job scheduling method in this specification; Figure 4 The corresponding Figure 3 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0023] Figure 1 is a schematic diagram of the structure of a job scheduling system in this specification, wherein the system includes a task processing module and a high-performance computing module; The task processing module is used to receive large-scale computing tasks delivered by users, determine the task structure corresponding to the large-scale computing tasks, split the large-scale computing tasks according to the task structure and construct a number of computing jobs; send the computing jobs to the high-performance computing module; the high-performance computing module is used to receive and process the computing jobs sent by the task processing module, and the high-performance computing module includes a number of computing cluster modules.

[0024] The job scheduling system provided in this specification can be used for scheduling and processing any computing tasks, such as large-scale computing tasks, homogeneous computing tasks, etc. In this specification, the processing of large-scale computing tasks is mainly used as an example for explanation.

[0025] In the job scheduling system provided in this specification, the task processing module can be mainly used to interact with users and split and reorganize computing tasks. Generally, job scheduling starts with receiving computing tasks. Generally, large-scale computing tasks are delivered to the job scheduling system by users, and the task processing module can first receive and store large-scale computing tasks delivered by users. At the same time, the task processing module can also obtain resource information of each high-performance computing module, wherein the resource information of a high-performance computing module can be the total amount of computing resources of this high-performance computing module and the amount of computing resources currently available. For the received large-scale computing tasks, the task processing module can split and reorganize the large-scale computing tasks into several computing jobs and send them to the high-performance computing module. There can be multiple high-performance computing modules, and the functions of each high-performance computing module are the same, all of which are used to process the computing jobs sent by the task processing module. There can be multiple structures of high-performance computing modules, for example, high-performance computing modules can be constructed based on workload management clusters such as Slurm and Kubernetes, and this specification does not make specific restrictions on this. There are several computing cluster modules in each high-performance computing module for providing computing power and processing computing jobs.

[0026] The main function of the job scheduling system provided in this specification is to split the received large-scale computing tasks and reorganize them into smaller-scale computing jobs, so that the load of the large-scale computing tasks is distributed to different high-performance computing modules, increasing resource utilization and task processing efficiency. The splitting of large-scale computing tasks depends on the configuration information of the large-scale computing tasks themselves. Therefore, when receiving large-scale computing tasks, the configuration information of the large-scale computing tasks can be specifically parsed, and the configuration information includes the name of the large-scale computing tasks and several task data unit parameters, and the task data unit parameters include but are not limited to text input, file selection, and directory; store the configuration information, and set the state of the large-scale computing tasks to the running state.

[0027] The configuration information of a large-scale computing task may include the name of the large-scale computing task and the singular number of the task data unit, wherein the expression form of the task data unit parameter may exist in multiple forms, including but not limited to text input, file selection, directory, etc. After the configuration information of the large-scale computing task is successfully stored, the large-scale computing task can be split based on the configuration information. Specifically, the task structure can be determined according to the configuration information contained in the large-scale computing task, wherein the task structure is jointly described by the minimum computing task and a number of task data units, wherein the minimum computing task is used to characterize the execution process, input and output and execution environment of the large-scale computing task, and the task data unit includes a general task data unit and a key task data unit, wherein the general task data unit represents data that has no effect on the computing resources required for the actual computing task generated by the task structure, and the key task data unit represents data that has an effect on the computing resources required for the actual computing task generated by the task structure; wherein the minimum computing task, the general task data unit and the key task data unit are components for generating a computing job that can actually run; and the unit information of each task data unit is determined.

[0028] The task structure is used to describe the structure to be followed when executing large-scale computing tasks. In simple terms, it can be understood as the paradigm of the computing job to be executed. In this method, the task structure can be described as a minimum computing task and several task data units. Among them, the minimum computing task defines the execution process, input and output, and execution environment of the large-scale computing task. In other words, the minimum computing task is the template of the computing job finally generated. The task data unit is the data needed to execute the computing task. The task data unit can be divided into general data units and key data units. Among them, the general task unit is used to represent the data that has no effect on the computing resources, and the key data unit represents the data that has an effect on the computing resources. In simple terms, adding or reducing general data units in a computing job will not change the computing resources required to execute this computing job; conversely, adding or reducing key data units in a computing job will change the computing resources required to execute this computing job. The computing job finally generated can be described by the minimum computing task, general data units, and key data units. In other words, according to the computing job template specified by the minimum computing task, the required data is selected to form a computing job.

[0029] At the same time, the unit information of each data unit can be further determined. Specifically, the name, type, data value and data usage characteristics of each task data unit can be determined; the data value of the task data unit is decomposed according to the data usage characteristics to obtain a task data entry, and the task data entry is marked as unprocessed, wherein the data value of the general task data unit is decomposed to obtain a general task data entry, and the data value of the key task data unit is decomposed to obtain a key task data entry.

[0030] The unit information of each task data unit may include the name, type, data value and data usage characteristics of the data unit. Among them, the data usage characteristics are used to characterize how the data values ​​contained in the task data unit should be processed. A task data unit may contain multiple different data values. When building a computing job, several data values ​​in several task data units may be used, and not all data values ​​in a task unit will necessarily be used. Based on this, in this method, each data value can be used as a separate task data entry. When decomposing each data value, the data value of the general task data unit is determined as a general task data entry, and the data value of the critical task data unit is determined as a critical task data entry.

[0031] When constructing a computing job, it is also necessary to consider whether the current available computing resources of the high-performance computing module are sufficient to process the constructed computing job. Specifically, the minimum computing resources required for the task structure can be determined; the available computing resources of the high-performance computing module can be obtained; when the available computing resources are less than the minimum computing resources, the generation of this computing job is skipped; when the available computing resources are not less than the minimum computing resources, the number of key task data entries that can be processed simultaneously by the high-performance computing module is determined according to the minimum computing resources and the available computing resources; according to the number of key task data entries that can be processed simultaneously, no more than the number of key task data entries are read as key task data entries to be processed; according to the unit information of the key task data unit in the task structure and the key task data entry to be processed, the actual resource requirements of the computing job to be generated are analyzed; according to the data usage characteristics of the general task data unit in the task structure, the parameter value information of the general task data unit is read; according to the minimum computing task, the key task data entry to be processed, the parameter value information of the general task data unit, and the actual resource requirements, a job startup file is generated in a combined manner; and the computing job is constructed according to the job startup file.

[0032] For a pre-set task structure, the minimum computing resources required for the computing job constructed under this task structure can be determined. At the same time, the task processing module can monitor the information of the available computing resources of each high-performance computing module. If the available computing resources of the high-performance computing module are less than the minimum computing resources required for the computing job, it means that there are currently no available resources to process the computing job, and the generation of this computing job can be skipped directly.

[0033] On the contrary, if the available computing resources are not less than the minimum computing resources and are sufficient to process a certain amount of computing jobs, then we can further analyze how many task data items the current available computing resources can support the high-performance computing module to process simultaneously. Since general task data units are data that have no effect on the computing resources required for computing jobs, when constructing computing jobs, we need to determine how many key task data items the available computing resources can process simultaneously.

[0034] After determining the number of critical task data entries that can be processed, the corresponding number of critical task data entries can be read as pending critical task data entries to be processed. At the same time, the computing resources required for each critical task data unit can be known through the unit information of the critical task data unit, and the actual resource requirements of the computing job to be generated can be further determined. At the same time, according to the data usage characteristics of the general data unit, the parameter values ​​of the required general task data units can be read in conjunction with the read critical task data entries. In this way, a job startup file can be generated based on the acquired critical task data entries, the parameter value information of the general task data unit, and the actual resource requirements. The job startup file is a constructed code file that can automatically construct a computing job that meets its description when it is run.

[0035] Preferably, in the job scheduling system provided in the specification, each high-performance computing module may include several computing cluster modules. When constructing the task data entry into a computing job according to the resource information of the high-performance computing module, the available computing resources of each computing cluster module may be specifically determined for each computing cluster module; and computing jobs matching the computing cluster modules may be constructed according to the available computing resources of each computing cluster module and the computing resources required for processing the task data entry.

[0036] In order to make the resource allocation in the high-performance computing module more reasonable, each high-performance computing module can have multiple different computing cluster modules to enhance the high-performance computing module and its ability to process tasks. Each computing cluster module can be used independently to process computing jobs, and the size of the computing resources owned by the computing cluster module can be set according to specific needs. The size of the computing resources owned by each computing cluster module can be the same or different, and this specification does not impose specific restrictions on this.

[0037] Since after the task processing module assigns the computing job to the high-performance computing module, the computing cluster module in the high-performance computing module actually processes the computing job, the available computing resources in each computing cluster module need to be considered when constructing the computing job. The available computing resources of a computing cluster module refer to the amount of idle computing resources in this computing cluster module when constructing the computing job. By combining the available computing resources of each computing cluster module with the computing resources required to process each task data entry, the task data entry can be reasonably constructed into a computing job.

[0038] The computing cluster module in each high-performance computing module actually processes the computing job, and each computing job is matched with the computing cluster module when constructing the computing job. Therefore, when the task processing module interacts with the high-performance computing module, each computing job needs to be sent to the high-performance computing module to which the computing cluster module that matches the computing job belongs, so that the high-performance computing module can send the received computing job to the matching computing cluster module for processing.

[0039] In addition, one point that needs to be considered in the process of job scheduling is that any computing job has the possibility of exceptions during the processing. There are many reasons for the exceptions during processing, such as data transmission errors, calculation errors, etc. When the computing cluster module processes the computing job abnormally, it is obviously impossible to obtain effective computing results. To solve this problem, in the job scheduling system provided in this specification, the task processing module can also be used to periodically monitor the running status of the high-performance computing module; in response to the completion of any computing job executed by any high-performance computing module, the status of the key task entry contained in the computing job is marked as completed; in response to any high-performance computing module in the processing of the computing job abnormal, the task data entry contained in the abnormal computing job is restored, the restored key task data entry is stored in the key task data unit data entry library, the general task data entry is stored in the general task data unit data entry library, and the status of the restored task data entry is marked as unfinished; in response to all the key task data entries contained in the large-scale computing task being in a completed state, the status of the large-scale computing task is marked as completed.

[0040] The task processing module can monitor the running status of all high-performance computing modules, wherein the running status of the high-performance computing modules may include whether each computing cluster module is running, whether an exception occurs, etc. Thus, when an exception occurs in a computing cluster module within a high-performance computing module when processing a computing job, the task processing module can discover the exception at the first time through monitoring. In the case that there is no exception in a computing job, the task processing module will restore the computing job processed by this computing cluster module to a task data entry, obtain key task data entries and general task data entries respectively, and store them in the corresponding entry library respectively, and mark the restored task data entries as unfinished again, and reconstruct these task data entries into computing jobs in the next subsequent job scheduling. It should be noted that there is no correlation between each job scheduling, and the restored task data entries may still be constructed together as the original computing job in the next job scheduling, or they may be merged with other task data entries to construct different computing jobs.

[0041] When all key task data entries have been processed and are in a completed state, it can be considered that the entire large-scale computing task has been processed and completed, and the large-scale computing task can be marked as completed.

[0042] It is conceivable that, for large-scale computing tasks, after performing one job scheduling in the above manner in the job scheduling system provided in this specification, all the calculations in the large-scale computing task may not be completed. Therefore, the job scheduling system provided in this specification can perform multiple job scheduling for a large-scale computing task cycle in the above manner. Specifically, the job scheduling system can repeatedly obtain resource information of each high-performance computing module, construct the task data entry into a computing job according to the resource information, and send the computing job to the high-performance computing module.

[0043] At the same time, the high-performance computing module can also be used to monitor the resource information and hardware status of each computing cluster module contained in it, and report to the task processing module when interacting with the task processing module. After a job scheduling allocates computing jobs to all computing cluster modules, if there are still remaining task data entries obtained by splitting large-scale computing tasks, the task processing module can wait until the computing job is processed, or wait for a certain period of time according to a preset cycle, and then obtain the resource information of each high-performance computing module again, construct the remaining task data entries into computing jobs and send them to the high-performance computing module for processing until all task data entries are processed. Among them, the remaining task data entries may include task data entries that are restored after an exception occurs during the processing of the computing cluster module during the last job scheduling process.

[0044] Additionally, the job scheduling system provided in this specification may also include a data management module for accessing the data uploaded by the user, storing the data in a structured and / or unstructured manner, and adding, deleting, rewriting, and querying the data.

[0045] The job scheduling system provided in this specification can split the received large-scale computing tasks through the task processing module, and construct the split task data entries into computing jobs that match the resources of the high-performance computing modules according to the resource information of each high-performance computing module, and the high-performance computing modules process the computing jobs. The system can be used to decompose complex large-scale tasks into task data entries and construct them into computing jobs that match the resources of the high-performance computing modules, so that the load brought by the large-scale computing tasks can be reasonably distributed to each high-performance computing module, effectively improving the utilization rate of computing resources and the processing efficiency of computing tasks.

[0046] Additional, such as Figure 1 As shown, in the job scheduling system provided in this specification, there may also be a network file module. The network file module can be used to receive and store the job information sent by the task processing module, the job information at least including each computing job and the task data items contained in each computing job, receive and store the calculation results of the computing job sent by the high-performance computing module, and return the calculation results to the user after matching the computing job.

[0047] The network file module is mainly used to implement network-related functions in the job scheduling system. When the task processing module completes the definition and generation of the computing job, the job information of the computing job can be sent to the network file module for storage. The job information may include, for example, an identifier for uniquely identifying each computing job, and information about the task data entries contained in each computing job. When the high-performance computing module completes the processing of the computing job, the computing result can be sent to the network file module. After the network file module matches the received computing result with the stored job information of the computing job, it returns the computing result correctly to the user.

[0048] In addition, the network file module can also be used to implement network management, configuration initialization and other functions for other modules in the job scheduling system. Specifically, the network file module can use the specified network protocol to mount network files for the task processing module and the high-performance computing module, so that the task processing module and the high-performance computing module can realize network functions; at the same time, the network file module can install the computing framework and operating environment required for processing computing jobs for each high-performance computing module.

[0049] Figure 2 This is a specific structural diagram of a task processing module of a job scheduling system provided in this specification. Figure 2 As shown, the task processing module may include a cluster communication module, a task management module, an operation monitoring module, a network connection module, and a data storage module.

[0050] like Figure 2 As shown, the cluster communication module can be used to interact with each high-performance computing module and send computing jobs to each high-performance computing module. The task management module can be used to interact with users, receive computing tasks submitted by users, split computing tasks into task data entries, and reorganize task data entries into computing jobs. The operation monitoring module is used to monitor the operating status of each high-performance computing module in real time, obtain resource information of each high-performance computing module, release the computing resources of the computing cluster module when each computing cluster module completes the processing of the computing job, and at the same time find out whether there is an abnormal computing cluster module. The network connection module is used to interact with the network file module, mount the network file system, and enable the task processing module to implement network-related functions. The data storage module is used to dimension the data generated in the process of job scheduling, including but not limited to task delivery data, task data entry data, computing job data, high-performance computing module status data, etc.

[0051] The above is the job scheduling system provided in this manual. Based on the same idea, this manual also provides corresponding job scheduling methods, such as Figure 3 shown.

[0052] Figure 3 This is a process diagram of a job scheduling method provided in this specification. The method is applied to a task processing module in a job scheduling system provided in this specification. The method specifically includes: S100: receiving large-scale computing tasks delivered by users; S102: Determine a task structure corresponding to the large-scale computing task; S104: splitting the large-scale computing task according to the task structure and constructing a plurality of computing jobs; S106: Send the computing job to a high-performance computing module.

[0053] The above-mentioned job scheduling method corresponds to the job scheduling system provided in this specification. Its specific implementation has been explained in detail in the job scheduling system provided in this specification, and this specification will not repeat it here.

[0054] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 3 Provides job scheduling methods.

[0055] This manual also provides Figure 4The schematic structure diagram of the electronic device shown in FIG. Figure 4 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 3 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0056] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0057] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0058] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0059] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0060] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0061] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0064] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0065] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0066] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0067] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0068] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0070] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0071] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A job scheduling system, characterized in that: The system comprises: a task processing module and a high-performance computing module; The task processing module is used to receive the large-scale computing task delivered by the user, determine the task structure corresponding to the large-scale computing task, split the large-scale computing task according to the task structure and construct a plurality of computing jobs; and send the computing job to the high-performance computing module; The high-performance computing module is used to receive and process the computing jobs sent by the task processing module. The high-performance computing module includes several computing cluster modules.

2. The system according to claim 1, characterized in that Receive large-scale computing tasks delivered by users, including: Parsing configuration information of the large-scale computing task, the configuration information including the name of the large-scale computing task and several task data unit parameters, the task data unit parameters including but not limited to text input, file selection, and directory; The configuration information is stored, and the state of the large-scale computing task is set to a running state.

3. The system according to claim 2, characterized in that Determine a task structure corresponding to the large-scale computing task, specifically including: Determine a task structure according to the configuration information contained in the large-scale computing task, wherein the task structure is jointly described by a minimum computing task and a plurality of task data units, wherein the minimum computing task is used to characterize the execution process, input and output, and execution environment of the large-scale computing task, and the task data units include general task data units and key task data units, wherein the general task data units represent data that have no effect on the computing resources required for the actual computing task generated by adopting the task structure, and the key task data units represent data that have an effect on the computing resources required for the actual computing task generated by adopting the task structure; wherein the minimum computing task, the general task data units, and the key task data units are components for generating a computing job that can actually be run; Determine the unit information of each task data unit.

4. The system according to claim 3, characterized in that Determine the unit information of each task data unit, including: Determine the name, type, data value and data usage characteristics of each task data unit; The data value of the task data unit is decomposed according to the data usage characteristics to obtain task data entries, and the task data entries are marked as unprocessed, wherein the data value of the general task data unit is decomposed to obtain the general task data entry, and the data value of the critical task data unit is decomposed to obtain the critical task data entry.

5. The system according to claim 4, characterized in that The large-scale computing task is split according to the task structure and a number of computing jobs are constructed, specifically including: determining the minimum computing resources required for the task structure; Obtaining available computing resources of the high-performance computing module; When the available computing resources are less than the minimum computing resources, skip the generation of this computing job; when the available computing resources are not less than the minimum computing resources, determine the number of key task data items that can be processed simultaneously by the high-performance computing module according to the minimum computing resources and the available computing resources; According to the number of the key task data entries that can be processed simultaneously, reading the key task data entries not exceeding the number as the key task data entries to be processed; parsing actual resource requirements of the computing job to be generated according to the unit information of the key task data unit in the task structure and the key task data entry to be processed; Reading parameter value information of the general task data unit according to the data usage characteristics of the general task data unit in the task structure; Generate a job startup file in a combined manner according to the minimum computing task, the key task data entry to be processed, the parameter value information of the general task data unit, and the actual resource demand; The computing job is constructed according to the job startup file.

6. The system according to claim 1, wherein: The task processing module is also used to periodically monitor the operating status of the high-performance computing module; In response to completion of any computing job executed by any high-performance computing module, marking the status of a key task item included in the computing job as completed; In response to an exception occurring in any high-performance computing module while processing a computing job, the task data entries included in the computing job in which the exception occurred are restored, the restored key task data entries are stored in a key task data unit data entry library, the general task data entries are stored in a general task data unit data entry library, and the status of the restored task data entries is marked as incomplete; In response to all the key task data items included in the large-scale computing task being in a completed state, marking the state of the large-scale computing task as completed.

7. The system according to claim 1, characterized in that The high-performance computing module is also used to monitor the resource information and hardware status of each computing cluster module included in the high-performance computing module.

8. The system of claim 1, wherein: The system also includes a data management module for accessing the data uploaded by the user, storing the data in a structured and / or unstructured manner, and performing addition, deletion, rewriting and query operations on the data.

9. A job scheduling method, characterized in that: The method is applied to a task processing module of a job scheduling system according to any one of claims 1 to 8, and the method comprises: Receive large-scale computing tasks submitted by users; Determining a task structure corresponding to the large-scale computing task; Splitting the large-scale computing task according to the task structure and constructing a plurality of computing jobs; The computing operation is sent to a high performance computing module.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to claim 9 is implemented.

Images