A parallel computing system resource scheduling method and system for big data processing

Abstract

This invention relates to the field of data processing technology and discloses a resource scheduling method and system for parallel computing systems oriented towards big data processing. The method involves real-time acquisition of raw state data from computing nodes in a parallel computing cluster, preprocessing and sharding the raw state data, dynamically adjusting the sharding granularity based on data correlation, and obtaining a fine-grained state vector sequence. This fine-grained state vector sequence is then input into an LSTM-Transformer model for temporal feature extraction and fusion to predict resource demands within future time windows, outputting the prediction results. A multi-agent proximal policy optimization framework is constructed, with each agent corresponding to a computing node. Based on the prediction results, the multi-agent proximal policy optimization framework generates a set of candidate scheduling schemes, and Pareto front analysis is used to select the optimal scheduling scheme, which is then distributed to the computing nodes for execution. This invention improves resource scheduling efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a resource scheduling method and system for parallel computing systems oriented towards big data processing. Background Technology

[0002] With the rapid development of big data technology, the amount of data generated by various application scenarios is growing exponentially. Parallel computing systems, with their multi-node collaborative processing capabilities, have become the core support for efficient big data processing. Resource scheduling, as a core component of parallel computing systems, is responsible for rationally allocating massive data processing tasks to various computing nodes and coordinating the allocation and utilization of hardware resources. Traditional resource scheduling methods are difficult to adapt to complex and ever-changing operating conditions. Existing scheduling methods mostly rely on fixed allocation rules and simple data statistics to conduct resource assessment, which can only collect single-dimensional resource data and cannot complete the fine-grained analysis of multi-dimensional resource status. Traditional scheduling architectures mostly adopt a centralized decision-making model, with weak node autonomous scheduling capabilities. It is difficult to balance global coordination and local adaptation, and there is a lack of intelligent collaborative decision-making mechanisms. This easily leads to problems such as resource idleness and waste, task queuing and congestion, and node load imbalance. The overall scheduling flexibility and adaptability are poor, which cannot meet the actual needs of efficient and stable operation of large-scale big data parallel computing. Summary of the Invention

[0003] The purpose of this invention is to solve the above-mentioned problems by designing a resource scheduling method and system for parallel computing systems oriented towards big data processing.

[0004] The first aspect of this invention provides a resource scheduling method for a parallel computing system oriented towards big data processing, the method comprising the following steps: The raw state data of computing nodes in the parallel computing cluster is collected in real time, preprocessed and sharded, and the sharding granularity is dynamically adjusted based on the data correlation to obtain a fine-grained state vector sequence. The fine-grained state vector sequence is input into the LSTM-Transformer model for temporal feature extraction and fusion, predicting resource demand within the future time window and outputting the prediction results. Construct a multi-agent proximal policy optimization framework that includes multiple agents, with each agent corresponding to a computing node; Based on the prediction results, a multi-agent near-end strategy optimization framework is used to generate a set of candidate scheduling schemes. Pareto front analysis is then used to screen the optimal scheduling scheme, which is then distributed to the computing nodes for execution.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the real-time acquisition of raw state data of computing nodes in the parallel computing cluster, preprocessing and sharding the raw state data, and dynamically adjusting the sharding granularity based on data correlation to obtain a fine-grained state vector sequence includes: The raw state data of all computing nodes in the parallel computing cluster is collected in real time. The collected raw state data is processed by removing outliers and filling missing values ​​to obtain cleaned state data. Determine the number of partitions K, randomly select K data samples as initial cluster centers, use the K-Means algorithm to partition the cleaned state data, and output the clustering partitioning results; Based on the clustering and sharding results, the correlation between different resource indicator data in each shard is calculated. If the correlation is higher than a preset threshold range, the current shard is split. If the correlation is lower than the preset threshold range, adjacent shards are merged. Finally, a fine-grained state vector sequence including CPU utilization, memory usage, network throughput, and disk I / O is obtained.

[0006] Optionally, in a second implementation of the first aspect of the present invention, determining the number of partitions K, randomly selecting K data samples as initial cluster centers, using the K-Means algorithm to partition the cleaned state data, and outputting the clustering partitioning results includes: Calculate the Euclidean distance between each data sample in the cleaned state data and each initial cluster center, and assign each data sample to the partition containing the nearest cluster center to complete the first cluster assignment; The mean of each segment is recalculated, and the mean is used as the new cluster center. The cluster centers are continuously updated until the preset number of iterations is reached to obtain the clustering segmentation results.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the step of inputting the fine-grained state vector sequence into the LSTM-Transformer model for temporal feature extraction and fusion, predicting resource demand within a future time window, and outputting the prediction result includes: The fine-grained state vector sequence is sorted in chronological order and then input into the LSTM-Transformer model; The input fine-grained state vector sequence is first processed by the LSTM network. The resource state data of each time step is read sequentially, the short-term temporal dependency features in the sequence are captured, and the local feature information of each time step is extracted to obtain the short-term temporal features. Short-term temporal features are input into the Transformer encoder. The Transformer encoder uses a multi-head attention mechanism to compute the correlation between different time steps and different resource indicators in parallel, capturing long-term temporal dependency features in the sequence to obtain long-term temporal features. By fusing short-term and long-term time-series features, a comprehensive time-series feature is obtained. Based on this comprehensive time-series feature, the Transformer encoder makes predictions on resource requirements within a preset future time window, generating prediction results including CPU demand prediction, memory demand prediction, task arrival rate prediction, and task execution time prediction.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the construction of a multi-agent proximal policy optimization framework comprising multiple agents, each agent corresponding to a computing node, includes: Based on the number of computing nodes in the parallel computing cluster, a corresponding number of agents are set up, and a local observation module is configured for each agent. The observation scope of the local observation module includes the real-time resource consumption of the node, the queue of tasks to be executed, and the task priority. Construct an Actor network for each agent. The Actor network outputs scheduling decision actions for computing nodes based on the local observation information of the agents. A centralized Critic network is built. The Critic network receives local observation information and decision actions from all agents, evaluates the global scheduling effect, and generates a global value function. Collect the experience data generated by each agent during the decision-making process, prioritize the experience data, and establish an experience sharing pool among agents.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of generating a set of candidate scheduling schemes based on the prediction results using a multi-agent proximal strategy optimization framework, screening the optimal scheduling scheme using Pareto front analysis, and distributing the optimal scheduling scheme to the computing nodes for execution includes: In the multi-agent proximal policy optimization framework, each agent makes independent scheduling decisions and generates its own local scheduling scheme based on the local observation information of its corresponding computing node and the input prediction results through the Actor network. The centralized Critic network performs a global evaluation of the local scheduling schemes generated by all agents, and adjusts and optimizes the local scheduling schemes by combining the global value function. It integrates all local scheduling schemes to generate multiple candidate scheduling schemes and forms a set of candidate scheduling schemes. Pareto front analysis is used to screen the candidate scheduling scheme set to obtain the optimal scheduling scheme. The optimal scheduling scheme is then decomposed into the execution instructions corresponding to each computing node and sent to the corresponding computing node.

[0010] Optionally, in the sixth implementation of the first aspect of the present invention, the step of using Pareto front analysis to screen the candidate scheduling scheme set to obtain the optimal scheduling scheme, and decomposing the optimal scheduling scheme into execution instructions corresponding to each computing node, and issuing them to the corresponding computing nodes, includes: Obtain the optimization objective of the scheduling scheme, wherein the optimization objective includes at least maximizing resource utilization and minimizing task execution latency; Calculate the performance index of each candidate scheduling scheme in the candidate scheduling scheme set under the optimization objective. Based on the Pareto front analysis results, select the scheduling schemes corresponding to the non-dominated solutions to obtain the candidate set of the optimal scheduling scheme. From the set of optimal scheduling schemes, and in combination with the requirements of the parallel computing system, the final optimal scheduling scheme is determined.

[0011] A second aspect of the present invention provides a resource scheduling system for parallel computing systems oriented towards big data processing, the system comprising: The data processing module is used to collect raw state data of computing nodes in the parallel computing cluster in real time, preprocess and shard the raw state data, dynamically adjust the sharding granularity based on the data correlation, and obtain a fine-grained state vector sequence. The feature extraction module is used to input fine-grained state vector sequences into the LSTM-Transformer model for temporal feature extraction and fusion, predict resource demand within future time windows, and output prediction results. The building module is used to construct a multi-agent proximal policy optimization framework that includes multiple agents, with each agent corresponding to a computing node; The scheme distribution module is used to generate a set of candidate scheduling schemes based on the prediction results using a multi-agent proximal strategy optimization framework, and to use Pareto front analysis to screen the optimal scheduling scheme, and then distribute the optimal scheduling scheme to the computing nodes for execution.

[0012] A third aspect of the present invention provides a resource scheduling device for a parallel computing system oriented towards big data processing. The resource scheduling device for a parallel computing system oriented towards big data processing includes a memory and at least one processor. The memory stores instructions. The at least one processor invokes the instructions in the memory to cause the resource scheduling device for a parallel computing system oriented towards big data processing to perform the steps of the resource scheduling method for a parallel computing system oriented towards big data processing as described in any of the preceding claims.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the resource scheduling method for a parallel computing system for big data processing as described in any of the preceding claims.

[0014] The technical solution provided by this invention involves real-time acquisition of raw state data from computing nodes in a parallel computing cluster. This raw state data is preprocessed and sharded, and the sharding granularity is dynamically adjusted based on data correlation to obtain a fine-grained state vector sequence. This fine-grained state vector sequence is then input into an LSTM-Transformer model for temporal feature extraction and fusion to predict resource demands within future time windows, outputting the prediction results. A multi-agent proximal policy optimization framework is constructed, with each agent corresponding to a computing node. Based on the prediction results, the multi-agent proximal policy optimization framework generates a set of candidate scheduling schemes, and Pareto front analysis is used to select the optimal scheduling scheme. The optimal scheduling scheme is then distributed to the computing nodes for execution. This invention effectively reduces task execution time, improves the overall utilization rate of various hardware and software resources in the cluster, enhances the adaptive scheduling capability of the parallel computing system, ensures the smooth and efficient advancement of big data processing services, and improves resource scheduling efficiency. Attached Figure Description

[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0016] Figure 1 A flowchart illustrating a resource scheduling method for a parallel computing system oriented towards big data processing, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the resource scheduling system for a parallel computing system oriented towards big data processing, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of a resource scheduling device for a parallel computing system oriented towards big data processing, provided in an embodiment of the present invention. Detailed Implementation

[0017] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0018] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 A flowchart of a resource scheduling method for a parallel computing system for big data processing provided in this embodiment of the invention is shown. The method specifically includes the following steps: Step 101: Collect raw state data of computing nodes in the parallel computing cluster in real time, preprocess and shard the raw state data, dynamically adjust the sharding granularity based on the data correlation, and obtain a fine-grained state vector sequence. In this embodiment, various raw operating status data generated by all computing nodes within the parallel computing cluster are continuously collected. All resource-related data information generated during node operation is comprehensively collected and uniformly summarized to form a complete raw dataset. Then, data screening is carried out on the summarized raw status data in sequence. Abnormal values ​​are removed and missing data positions are filled in uniformly. Data format and data standards are standardized and unified. Finally, all basic data standardization work is completed to obtain clean cluster node status data with overall data quality meeting the standards and without errors or gaps. The number of clustering segments K is determined according to the overall operating scale of the cluster and the total amount of data. K independent data samples are randomly selected from the overall data range that has been cleaned. The selected data samples are uniformly set as the initial clustering centers of the algorithm. Each independent data sample in the clean state data is traversed in turn. The distance relationship between a single data sample and all the set initial clustering centers is calculated one by one. The category is determined according to the distance. Each data sample is assigned to the segment group corresponding to the nearest clustering center, thus completing the first round of overall data clustering and grouping. After the first round of data grouping and allocation is completed, the overall average value of all included data samples within each independent partition group is uniformly calculated. The calculated average value of each group replaces the original cluster center and serves as the new cluster center to continue data classification calculation. The data distance calculation and group re-division process is continuously and repeatedly executed, and the cluster center iteration and update operation is completed without interruption. The entire process is carried out in a loop according to the established operation process until the overall operation process reaches the number of iterations preset by the system. The clustering operation process is then terminated and the complete and standardized overall data clustering and partitioning processing results are output. Based on the completed clustering and sharding results, correlation calculations are performed on the various types of resource indicators contained in each independent shard. The degree of data correlation between different resource indicators within a shard is accurately calculated. The results are strictly judged against the pre-set threshold range of correlation values. When the degree of data correlation within a shard is higher than the preset threshold range, the current shard is split and refined. When the degree of data correlation within a shard is lower than the preset threshold range, the shard is merged and integrated with adjacent shards. After multiple sharding adjustments and optimizations, a complete and standardized fine-grained resource state vector sequence is finally generated, covering CPU utilization, memory usage, network throughput, and disk read / write input / output status.

[0019] Step 102: Input the fine-grained state vector sequence into the LSTM-Transformer model for temporal feature extraction and fusion, predict resource demand within the future time window, and output the prediction results; In this embodiment, the sorted fine-grained state vector sequence is arranged in chronological order to clarify the logical relationship between the time sequence of all data samples, ensuring that the overall data time sequence arrangement is neat and error-free. The complete vector sequence after time sequence sorting is then fed into the constructed LSTM-Transformer combined model to complete the data input process of the model. The LSTM network inside the model performs data processing, reading the fine-grained state vector sequence one by one in a predetermined time sequence, retrieving the resource state-related data of each node corresponding to each time node in turn, and relying on its own operation logic to capture the short-term time sequence change correlation patterns within the entire sequence. The local resource feature content corresponding to each independent time node is extracted one by one. After the complete operation and extraction process, the final output is the formed short-term time sequence feature data content. The extracted short-term temporal features are fed into the Transformer encoder, which uses its internal multi-head attention mechanism to perform multi-dimensional parallel computations. It calculates and analyzes the intrinsic relationships between different time points and between different categories of resource indicators, uncovering long-term temporal dependencies hidden within the entire data sequence. After extracting long-term features, the corresponding long-term temporal features are obtained. The previously acquired short-term temporal features are then integrated and fused with the extracted long-term temporal features, combining all data information from both types to form a comprehensive temporal feature set. Based on this comprehensive temporal feature set and a pre-defined future prediction time window, the Transformer encoder performs demand estimation, sequentially analyzing multiple aspects to ultimately generate multi-dimensional prediction data results, including CPU resource demand prediction, memory resource demand prediction, task arrival rate prediction, and overall task execution time prediction.

[0020] Step 103: Construct a multi-agent proximal policy optimization framework that includes multiple agents, with each agent corresponding to a computing node; In this embodiment, based on the total number of computing nodes actually deployed within the parallel computing cluster, an equal number of decision-making agents are configured and deployed one by one to ensure that each computing node can be matched with a dedicated independent agent. At the same time, all deployed agents are uniformly equipped with matching local observation modules, and the data collection and information perception range of the observation modules is clearly defined, enabling them to perceive in real time the current resource occupancy of the corresponding node, the queue of tasks to be processed within the node, and the priority levels of various tasks, as well as other on-site operational information. After completing the basic configuration of the agents, a dedicated Actor decision network structure is built for each independent agent. During operation, this network uses all on-site observation information collected by the agent through the local observation module as the core input, relies on internal computing logic to complete information analysis and judgment, and outputs various resource scheduling and task allocation related decision actions adapted to the current operating status of the computing node according to the actual operating status of the node. A centralized Critic evaluation network with global overall assessment capabilities is established. This network can simultaneously receive local observation data uploaded and aggregated by all agents within the cluster, as well as scheduling decision actions output by various agents. It performs a comprehensive evaluation of the overall scheduling performance from the perspective of the cluster's overall operation, and completes numerical quantitative analysis based on the comprehensive evaluation results. Finally, it calculates and generates a global value function that can measure the overall scheduling quality. It comprehensively collects and aggregates various operational experience data generated by each agent throughout the process of autonomously making scheduling decisions, and fully includes the corresponding state information, execution actions, and corresponding feedback information before and after the decision. The experience data is prioritized according to its contribution to model scheduling optimization. At the same time, an open and shared experience data storage pool is built for all agents to achieve unified collection and interoperability of decision-making experience from all agents.

[0021] Step 104: Based on the prediction results, a set of candidate scheduling schemes is generated using a multi-agent near-end strategy optimization framework. Pareto front analysis is then used to screen the optimal scheduling scheme, and the optimal scheduling scheme is distributed to the computing nodes for execution.

[0022] In this embodiment, during the operation of the multi-agent near-end strategy optimization framework, each independent agent retrieves all local observation information collected by its bound computing node, and simultaneously accesses the various resource demand prediction results obtained from previous calculations. Relying on its own Actor network, it independently conducts logical analysis and autonomous judgment, and independently completes the decision-making work related to its own scheduling, generating independent local scheduling execution schemes adapted to the actual operating conditions of its corresponding node. The centralized Critic network uniformly receives various local scheduling schemes formulated by all agents in the cluster, conducts a comprehensive global evaluation from the perspective of the overall cluster operation, and determines the merits of the schemes by combining the pre-formed global value function. According to the overall operating standards, it reasonably corrects and optimizes the local scheduling schemes with deviations, and integrates and summarizes all the adjusted local scheduling content in a unified manner, generating multiple scheduling candidate schemes with different arrangement forms in batches, and uniformly collecting and assembling them into a complete set of candidate scheduling schemes. The core optimization guidelines and evaluation criteria for this resource scheduling work were clearly defined in advance, and the optimization goals to be achieved were determined. These goals include, but are not limited to, comprehensively improving the utilization efficiency of all types of hardware resources in the cluster and reducing the overall execution waiting time and latency of various business tasks. This serves as the unified evaluation basis for subsequent solution selection. Quantitative calculations were then performed on each candidate scheduling solution within the candidate solution set, and the actual operational performance indicators of different candidate solutions under the defined optimization goals were statistically analyzed. Data comparison and selection were conducted strictly according to the judgment logic of Pareto front analysis. From this process, various scheduling schemes that meet the non-dominated solution determination criteria are selected. The selected schemes are then collected and integrated to form an optimal scheduling scheme candidate set. Considering the current overall operating load of the parallel computing system, the business task arrangement, and the actual on-site operating needs, the scheme with the strongest adaptability is selected from the optimal scheduling scheme candidate set as the final optimal scheduling scheme. The complete overall scheduling scheme is then broken down layer by layer according to the node division rules, decomposed into operation instructions that can be executed individually by each independent computing node, and finally distributed and delivered to the corresponding computing nodes one by one for execution.

[0023] Please see Figure 2 A schematic diagram of the resource scheduling system for a parallel computing system for big data processing provided in this embodiment of the invention. The system includes: The data processing module is used to collect raw state data of computing nodes in the parallel computing cluster in real time, preprocess and shard the raw state data, dynamically adjust the sharding granularity based on the data correlation, and obtain a fine-grained state vector sequence. The feature extraction module is used to input fine-grained state vector sequences into the LSTM-Transformer model for temporal feature extraction and fusion, predict resource demand within future time windows, and output prediction results. The building module is used to construct a multi-agent proximal policy optimization framework that includes multiple agents, with each agent corresponding to a computing node; The scheme distribution module is used to generate a set of candidate scheduling schemes based on the prediction results using a multi-agent proximal strategy optimization framework, and to use Pareto front analysis to screen the optimal scheduling scheme, and then distribute the optimal scheduling scheme to the computing nodes for execution.

[0024] Figure 3This is a schematic diagram of a resource scheduling device for a parallel computing system oriented towards big data processing, provided in an embodiment of the present invention. This resource scheduling device 300 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the resource scheduling device 300. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the resource scheduling device 300 to implement the method provided in the above embodiment.

[0025] The parallel computing system resource scheduling device 300 for big data processing may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The resource scheduling device structure of the parallel computing system for big data processing shown does not constitute a limitation on the computer device provided by the present invention. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0026] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the various steps of the parallel computing system resource scheduling method for big data processing provided in the above embodiments.

[0027] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described equipment or apparatus / unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0028] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0029] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A resource scheduling method for a parallel computing system oriented towards big data processing, characterized in that, The method includes the following steps: The raw state data of computing nodes in the parallel computing cluster is collected in real time, preprocessed and sharded, and the sharding granularity is dynamically adjusted based on the data correlation to obtain a fine-grained state vector sequence. The fine-grained state vector sequence is input into the LSTM-Transformer model for temporal feature extraction and fusion, predicting resource demand within the future time window and outputting the prediction results. Construct a multi-agent proximal policy optimization framework that includes multiple agents, with each agent corresponding to a computing node; Based on the prediction results, a multi-agent near-end strategy optimization framework is used to generate a set of candidate scheduling schemes. Pareto front analysis is then used to screen the optimal scheduling scheme, which is then distributed to the computing nodes for execution.

2. The resource scheduling method for a parallel computing system oriented towards big data processing as described in claim 1, characterized in that, The process involves real-time acquisition of raw state data from computing nodes in the parallel computing cluster, preprocessing and sharding the raw state data, dynamically adjusting the sharding granularity based on data correlation, and obtaining a fine-grained state vector sequence, including: The raw state data of all computing nodes in the parallel computing cluster is collected in real time. The collected raw state data is processed by removing outliers and filling missing values ​​to obtain cleaned state data. Determine the number of partitions K, randomly select K data samples as initial cluster centers, use the K-Means algorithm to partition the cleaned state data, and output the clustering partitioning results; Based on the clustering and sharding results, the correlation between different resource indicator data in each shard is calculated. If the correlation is higher than a preset threshold range, the current shard is split. If the correlation is lower than the preset threshold range, adjacent shards are merged. Finally, a fine-grained state vector sequence including CPU utilization, memory usage, network throughput, and disk I / O is obtained.

3. The resource scheduling method for a parallel computing system oriented towards big data processing as described in claim 2, characterized in that, The process involves determining the number of partitions K, randomly selecting K data samples as initial cluster centers, and using the K-Means algorithm to partition the cleaned state data, outputting the clustering partitioning results, including: Calculate the Euclidean distance between each data sample in the cleaned state data and each initial cluster center, and assign each data sample to the partition containing the nearest cluster center to complete the first cluster assignment; The mean of each segment is recalculated, and the mean is used as the new cluster center. The cluster centers are continuously updated until the preset number of iterations is reached to obtain the clustering segmentation results.

4. The resource scheduling method for a parallel computing system oriented towards big data processing as described in claim 1, characterized in that, The process involves inputting a fine-grained state vector sequence into an LSTM-Transformer model for temporal feature extraction and fusion, predicting resource demand within a future time window, and outputting the prediction results, including: The fine-grained state vector sequence is sorted in chronological order and then input into the LSTM-Transformer model; The input fine-grained state vector sequence is first processed by the LSTM network. The resource state data of each time step is read sequentially, the short-term temporal dependency features in the sequence are captured, and the local feature information of each time step is extracted to obtain the short-term temporal features. Short-term temporal features are input into the Transformer encoder. The Transformer encoder uses a multi-head attention mechanism to compute the correlation between different time steps and different resource indicators in parallel, capturing long-term temporal dependency features in the sequence to obtain long-term temporal features. By fusing short-term and long-term time-series features, a comprehensive time-series feature is obtained. Based on this comprehensive time-series feature, the Transformer encoder makes predictions on resource requirements within a preset future time window, generating prediction results including CPU demand prediction, memory demand prediction, task arrival rate prediction, and task execution time prediction.

5. A resource scheduling method for a parallel computing system oriented towards big data processing as described in claim 1, characterized in that, The construction includes a multi-agent proximal policy optimization framework comprising multiple agents, each agent corresponding to a computing node, including: Based on the number of computing nodes in the parallel computing cluster, a corresponding number of agents are set up, and a local observation module is configured for each agent. The observation scope of the local observation module includes the real-time resource consumption of the node, the queue of tasks to be executed, and the task priority. Construct an Actor network for each agent. The Actor network outputs scheduling decision actions for computing nodes based on the local observation information of the agents. A centralized Critic network is built. The Critic network receives local observation information and decision actions from all agents, evaluates the global scheduling effect, and generates a global value function. Collect the experience data generated by each agent during the decision-making process, prioritize the experience data, and establish an experience sharing pool among agents.

6. The resource scheduling method for a parallel computing system oriented towards big data processing as described in claim 5, characterized in that, The process involves generating a set of candidate scheduling schemes based on the prediction results using a multi-agent proximal strategy optimization framework, selecting the optimal scheduling scheme using Pareto front analysis, and then distributing the optimal scheduling scheme to the computing nodes for execution. This includes: In the multi-agent proximal policy optimization framework, each agent makes independent scheduling decisions and generates its own local scheduling scheme based on the local observation information of its corresponding computing node and the input prediction results through the Actor network. The centralized Critic network performs a global evaluation of the local scheduling schemes generated by all agents, and adjusts and optimizes the local scheduling schemes by combining the global value function. It integrates all local scheduling schemes to generate multiple candidate scheduling schemes and forms a set of candidate scheduling schemes. Pareto front analysis is used to screen the candidate scheduling scheme set to obtain the optimal scheduling scheme. The optimal scheduling scheme is then decomposed into the execution instructions corresponding to each computing node and sent to the corresponding computing node.

7. A resource scheduling method for a parallel computing system for big data processing as described in claim 6, characterized in that, The process involves using Pareto front analysis to screen the candidate scheduling scheme set, obtaining the optimal scheduling scheme, and then decomposing the optimal scheduling scheme into execution instructions corresponding to each computing node, which are then distributed to the corresponding computing nodes. This includes: Obtain the optimization objective of the scheduling scheme, wherein the optimization objective includes at least maximizing resource utilization and minimizing task execution latency; Calculate the performance index of each candidate scheduling scheme in the candidate scheduling scheme set under the optimization objective. Based on the Pareto front analysis results, select the scheduling schemes corresponding to the non-dominated solutions to obtain the candidate set of the optimal scheduling scheme. From the set of optimal scheduling schemes, and in combination with the requirements of the parallel computing system, the final optimal scheduling scheme is determined.

8. A resource scheduling system for parallel computing systems oriented towards big data processing, characterized in that, The system includes: The data processing module is used to collect raw state data of computing nodes in the parallel computing cluster in real time, preprocess and shard the raw state data, dynamically adjust the sharding granularity based on the data correlation, and obtain a fine-grained state vector sequence. The feature extraction module is used to input fine-grained state vector sequences into the LSTM-Transformer model for temporal feature extraction and fusion, predict resource demand within future time windows, and output prediction results. The building module is used to construct a multi-agent proximal policy optimization framework that includes multiple agents, with each agent corresponding to a computing node; The scheme distribution module is used to generate a set of candidate scheduling schemes based on the prediction results using a multi-agent proximal strategy optimization framework, and to use Pareto front analysis to screen the optimal scheduling scheme, and then distribute the optimal scheduling scheme to the computing nodes for execution.

9. A resource scheduling device for a parallel computing system oriented towards big data processing, characterized in that, The resource scheduling device for a parallel computing system oriented to big data processing includes a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the resource scheduling device for a parallel computing system oriented to big data processing to execute the various steps of the resource scheduling method for a parallel computing system oriented to big data processing as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the various steps of the resource scheduling method for parallel computing systems for big data processing as described in any one of claims 1-7.

Images