Data processing method and device, computer device and storage medium

Abstract

This application relates to a data processing method, apparatus, computer equipment, and storage medium. The method includes: acquiring a job and determining its target job type; querying a target database for the target job type's corresponding target weight, where the target weight is calculated based on the indicator value of the target performance event of a previous job of the same target job type; determining the target binding method corresponding to the target weight based on a preset correspondence between weights and binding methods; and binding the job based on the target binding method. By employing this method, the binding method of the job is determined based on the calculated weight of the current job. This allows for real-time adjustment of the current job's weight and its corresponding binding method based on the indicator values ​​of the performance events of preceding jobs, thereby ensuring the rationality of allocating computing resources to the job and improving job execution efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage systems, and particularly relates to a data processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] The SLURM (Simple Linux Utility Resource Management) system is an open source, fault-tolerant, high-scalability cluster job scheduling system. The SLURM can support management and allocation of multiple resources, and can query and archive multiple information generated in the running process of a job.

[0003] In the related art, a user can only configure a binding mode of a job in a node based on historical experience through the SLURM system. This node binding mode relying on historical experience leads to low efficiency of job running. SUMMARY

[0004] Therefore, it is necessary to provide a data processing method, device, computer equipment and storage medium capable of improving the running efficiency of a job.

[0005] In a first aspect, the present application provides a data processing method. The method comprises:

[0006] obtaining a job and determining a target job type of the job;

[0007] querying a target weight corresponding to the target job type in a target database, the target weight being calculated based on an index value corresponding to a target performance event of a job of a previous target job type;

[0008] determining a target binding mode corresponding to the target weight based on a preset correspondence between weights and binding modes, and binding the job based on the target binding mode.

[0009] In this embodiment, the weight of the current job is calculated based on the index value corresponding to the target performance event of the job of the previous target job type of the job, and the binding mode of the job is determined based on the calculated weight of the current job. The weight of the current job and the binding mode corresponding to the weight can be adjusted in real time based on the index value of the performance event of the previous job, so as to ensure the rationality of the allocation of computing resources for the job and improve the efficiency of the job running.

[0010] In one of the embodiments, the index value corresponding to the target performance event of the job of the previous target job type comprises a current performance index value, a historical performance index value, a current performance weight factor, a historical performance weight factor and a current weight value, and the method further comprises:

[0011] Calculate the current performance weighting factor based on the current performance index value and the historical performance index value;

[0012] The target weight is calculated based on the current performance weight factor, the weight factors of historical jobs, the preset weight, and the current weight value.

[0013] Update the weights of the target job type in the target database to the target weights.

[0014] In this embodiment, the weight value can be calculated based on the current performance index value and the historical performance index value to ensure the accuracy of the calculated weight value. Alternatively, the real-time performance of the calculated weight can be ensured by continuously updating the weights corresponding to different job types in the target database.

[0015] In one embodiment, calculating the target weight based on the current performance weight factor, the weight factors of historical jobs, a preset weight, and the current weight value includes:

[0016] Calculate the weight factor difference between the current performance weight factor and the weight factor of the historical job;

[0017] The weight adjustment factor is calculated based on the weight factor difference and the preset weight, and the current weight value is adjusted based on the weight adjustment factor to obtain the target weight.

[0018] In this embodiment, the current weight value is continuously adjusted by the weight factor difference degree, thereby achieving optimization of the target weight.

[0019] In one embodiment, after the step of binding the job based on the target binding method, the method further includes:

[0020] Collect the index values ​​corresponding to the target performance events of the job;

[0021] Calculate the first weight based on the index value corresponding to the target performance event of the task;

[0022] If, based on the target weight, it is determined that the first weight does not meet the preset convergence condition, the weight value of the target weight of the target job type in the target database is updated to the weight value of the first weight.

[0023] In this embodiment, the current weight value is continuously adjusted by the weight factor difference degree, thereby achieving optimization of the target weight.

[0024] In one embodiment, the data processing method further includes:

[0025] If the target weight corresponding to the target job type is not found in the target database, the default weight is determined to be the target weight corresponding to the target job type, and the step of determining the target binding method corresponding to the target weight based on the preset correspondence between weight and binding method is executed.

[0026] In this embodiment, the comprehensiveness of job type statistics can be ensured by pre-configuring the target weights of the initial jobs.

[0027] In one embodiment, the data processing method further includes:

[0028] Add the correspondence between the target job type and the target weight to the target database.

[0029] In this embodiment, by continuously updating the correspondence between job types and target weights contained in the target database, it is possible to ensure the integration and classification of job types and their corresponding weight values, thus providing a foundation for subsequent applications.

[0030] In one embodiment, collecting the metric values ​​corresponding to the target performance event of the job includes:

[0031] Create a performance event collection thread corresponding to the job;

[0032] If it is determined that no recycling information has been received from the main thread, the main thread and target performance event corresponding to the job are determined through the performance event collection thread corresponding to the job, and the indicator value of the target performance event is collected.

[0033] In this embodiment, by creating an independent performance event collection thread for each job, the collection of performance event corresponding indicator values ​​between different jobs can be decoupled, ensuring the efficiency of performance event collection.

[0034] Secondly, this application also provides a data processing apparatus. The apparatus includes:

[0035] The first acquisition module is used to acquire jobs and determine the target job type of the jobs;

[0036] The query module is used to query the target weight corresponding to the target job type in the target database. The target weight is calculated based on the index value corresponding to the target performance event of the previous target job type.

[0037] The first determining module is used to determine the target binding method corresponding to the target weight based on a preset correspondence between weights and binding methods, and to bind the job based on the target binding method.

[0038] In one embodiment, the metric value corresponding to the target performance event of the previous target job type includes the current performance metric value, historical performance metric value, current performance weight factor, historical performance weight factor, and current weight value. The device further includes:

[0039] The first calculation module is used to calculate the current performance weighting factor based on the current performance index value and the historical performance index value.

[0040] The second calculation module is used to calculate the target weight based on the current performance weight factor, the weight factor of historical jobs, the preset weight, and the current weight value.

[0041] The first update module is used to update the weight of the target job type in the target database to the target weight.

[0042] In one embodiment, the second computing module is specifically used for:

[0043] Calculate the weight factor difference between the current performance weight factor and the weight factor of the historical job;

[0044] The weight adjustment factor is calculated based on the weight factor difference and the preset weight, and the current weight value is adjusted based on the weight adjustment factor to obtain the target weight.

[0045] In one embodiment, the data processing apparatus further includes:

[0046] The data acquisition module is used to collect the index values ​​corresponding to the target performance events of the job.

[0047] The third calculation module calculates the first weight based on the index value corresponding to the target performance event of the job;

[0048] The second update module is used to update the weight value of the target weight of the target job type in the target database to the weight value of the first weight when it is determined based on the target weight that the first weight does not meet the preset convergence condition.

[0049] In one embodiment, the data processing apparatus further includes:

[0050] The second determining module is used to determine the default weight as the target weight corresponding to the target job type if the target weight corresponding to the target job type is not found in the target database, and to execute the step of determining the target binding method corresponding to the target weight based on the preset correspondence between weight and binding method.

[0051] In one embodiment, the data processing apparatus further includes:

[0052] An add module is used to add the correspondence between the target job type and the target weight to the target database.

[0053] In one embodiment, the acquisition module is specifically used for:

[0054] Create a performance event collection thread corresponding to the job; if it is determined that no recycling information is received from the main thread, determine the main thread and target performance event corresponding to the job through the performance event collection thread corresponding to the job, and collect the indicator value of the target performance event.

[0055] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0056] Obtain the job and determine the target job type of the job;

[0057] Query the target database for the target job type corresponding to the target weight, which is calculated based on the indicator value of the target performance event of the previous target job type.

[0058] Based on the preset correspondence between weights and binding methods, the target binding method corresponding to the target weight is determined, and the job is bound based on the target binding method.

[0059] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0060] Obtain the job and determine the target job type of the job;

[0061] Query the target database for the target job type corresponding to the target weight, which is calculated based on the indicator value of the target performance event of the previous target job type.

[0062] Based on the preset correspondence between weights and binding methods, the target binding method corresponding to the target weight is determined, and the job is bound based on the target binding method.

[0063] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0064] Obtain the job and determine the target job type of the job;

[0065] Query the target database for the target job type corresponding to the target weight, which is calculated based on the indicator value of the target performance event of the previous target job type.

[0066] Based on the preset correspondence between weights and binding methods, the target binding method corresponding to the target weight is determined, and the job is bound based on the target binding method.

[0067] The aforementioned data processing method, apparatus, computer equipment, and storage medium include: acquiring a job and determining its target job type; querying a target weight corresponding to the target job type in a target database, wherein the target weight is calculated based on the indicator value corresponding to the target performance event of the previous target job type; determining the target binding method corresponding to the target weight based on a preset correspondence between weights and binding methods; and binding the job based on the target binding method. By employing this method, the weight of the current job is calculated based on the indicator value corresponding to the target performance event of the previous target job type, and the binding method of the job is determined based on the calculated weight of the current job. This allows for real-time adjustment of the weight of the current job and the corresponding binding method based on the indicator values ​​of the performance events of preceding jobs, thereby ensuring the rationality of allocating computing resources to jobs and improving job execution efficiency. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating a data processing method in one embodiment;

[0069] Figure 2 This is a flowchart illustrating the target weight update step in one embodiment;

[0070] Figure 3 This is a flowchart illustrating the target weight update step in one embodiment;

[0071] Figure 4 This is a flowchart illustrating the step of calculating the first weight in one embodiment;

[0072] Figure 5 This is a flowchart illustrating the steps for collecting target performance event metric values ​​in one embodiment;

[0073] Figure 6a This is a flowchart illustrating a data processing method in one embodiment;

[0074] Figure 6b This is a flowchart illustrating the performance event collection thread in one embodiment;

[0075] Figure 7 This is a structural block diagram of a data processing device in one embodiment;

[0076] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0078] In one embodiment, such as Figure 1 As shown, a data processing method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc., and the server can be an independent server or a server cluster composed of multiple servers.

[0079] In this embodiment, the data processing method includes the following steps:

[0080] Step 102: Obtain the job and determine the target job type.

[0081] A job can be a task submitted by a user through a terminal, specifically a task submitted by a user on the SLURM system that requires the allocation of computing resources on a compute node. The target job type can be a classification of multiple jobs, such as a classification by the program to which the job corresponds, or a job that is a Matlab program, etc.

[0082] In practice, the terminal can respond to a job submission operation and obtain the job corresponding to the job submission operation. The terminal can also obtain the job corresponding to the job submission notification message after receiving the job submission notification message, etc. After obtaining the job, the terminal can determine the job type of the job, that is, the target job type.

[0083] Step 104: Query the target weight corresponding to the target job type in the target database.

[0084] The target weight is calculated based on the metric value corresponding to the target performance event of the previous target job type. The target database is a pre-configured database on the terminal that stores job types and their corresponding target weights. The target weights stored in this database are the weights to be updated. The previous target job type can be a job of the same type preceding the currently acquired target job type, arranged chronologically. A job performance event can be a software or hardware event that occurs during job execution and affects the performance of the program corresponding to the job. The target performance event of the job can be the performance event that has the greatest impact on the performance of the program corresponding to the job. The metric value corresponding to the performance event can be the value of that metric under the current conditions. For example, if the performance event represents CPU utilization, the metric value corresponding to the performance event is the specific value of the CPU utilization.

[0085] In implementation, after determining the target job type for the current job, the terminal can query the target database based on that target job. If the terminal can find the target job type in the target database, it can determine that a job of the same type as the current job has been executed. Based on this, the terminal can query the target weight corresponding to that target job type in the database. It should be noted that the calculation method for the job type and its corresponding target weight stored in the database can be as follows: after running the job, the terminal obtains the indicator value corresponding to the target performance event of the job, calculates the target weight corresponding to the job type based on the obtained indicator value, and updates the job type and its corresponding target weight in the target database.

[0086] In one embodiment, job performance events can include three types: Hardware Events, Software Events, and Kernel Tracepoint Events. Hardware Events are generated by the PMU (Power Management Unit) and detect whether and how many times performance events occur under specific conditions. Examples include CPU cycles, branch instructions, TLB refill exceptions, and cache misses. Software Events are events generated by the kernel and can be viewed using the `perf list` command. These low-level events based on kernel counters are distributed across various functional modules and are used to statistically analyze operating system-related performance events. Examples include the number of system calls, context switches, task migrations, and page faults. Kernel Tracepoint Events are events triggered by static tracepoints in the kernel. These tracepoints are used to determine the details of kernel behavior during program execution. For example, when a user uses `printk`, information in the kernel can be printed. The information to be printed can be selected at the debug level, i.e., the tracepoint.

[0087] Step 106: Based on the preset correspondence between weights and binding methods, determine the target binding method corresponding to the target weight, and bind the job based on the target binding method.

[0088] The pre-defined weight and binding method correspondence can be obtained by processing historical time period data in the terminal, and the binding method can be the binding method of the job within the computing node.

[0089] In implementation, the terminal can query the preset correspondence between weights and binding methods to determine the target binding method corresponding to the target weight of the job, and bind the job within the computing node corresponding to the job based on the target binding method. That is, based on the target binding method, computing resources are allocated to the job within the computing node, and the job is run based on the allocated computing resources.

[0090] In the above data processing method, jobs are acquired, and the target job type is determined. The target weight corresponding to the target job type is queried in the target database. The target weight is calculated based on the indicator value corresponding to the target performance event of the previous target job type. Based on the preset correspondence between weights and binding methods, the target binding method corresponding to the target weight is determined, and the job is bound based on the target binding method. By adopting this method, the weight of the current job is calculated based on the indicator value corresponding to the target performance event of the previous target job type, and the binding method of the job is determined based on the calculated weight of the current job. This allows for real-time adjustment of the weight of the current job and the corresponding binding method based on the indicator value of the performance event of the preceding job, thereby ensuring the rationality of allocating computing resources to the job and improving the efficiency of job execution.

[0091] In one embodiment, the metric values ​​corresponding to the target performance event of the previous target job type include the current performance metric value, historical performance metric values, current performance weight factor, historical performance weight factor, and current weight value.

[0092] Specifically, the current performance metric value represents the metric value obtained by the terminal collecting the metric corresponding to the target performance event after the previous target job type job has run; for clarity, the previous target job type job will be referred to as the first job in the following section; the historical performance metric value represents the metric value obtained by the terminal collecting the metric corresponding to the target performance event after the previous target job type job of the first job has run; similarly, the current performance weight factor represents the weight factor calculated based on the current performance metric value and historical performance metric value corresponding to the first job, and the historical performance weight factor represents the weight factor calculated based on the current performance metric value and historical performance value corresponding to the previous target job type job of the first job; the current weight value represents the target weight corresponding to the previous target job type job of the first job.

[0093] In one embodiment, the terminal can receive a second job and a third job of the same job type in chronological order. The second job can be the initial job of that job type received by the terminal. Upon receiving the second job, the terminal can use a default weight as the target weight for that second job and determine the target binding method corresponding to that target weight. The terminal can bind the job based on this target binding method, i.e., run the second job. While running the second job, the terminal collects the metric values ​​corresponding to the target performance events of the second job, calculates the weight (referred to as the first weight) based on the metric values ​​corresponding to the target performance events of the second job, and stores the job type of the second job (referred to as the target job type) and the first weight in the target database. After the terminal receives the third job, it can query the target database and use the first weight found as the target weight of the third job. Similarly, when the third job is running, the indicator values ​​corresponding to the target performance events of the third job are collected. Based on the indicator values ​​corresponding to the target performance events of the third job, the weight (referred to as the second weight) is calculated. The correspondence between the target job type and the first weight stored in the target database is updated to the correspondence between the target job type and the second weight. That is, the weight corresponding to the target job type is updated, which can be from the weight value of the first weight to the weight value of the second weight.

[0094] like Figure 2 As shown, the data processing method also includes:

[0095] Step 202: Calculate the current performance weight factor based on the current performance index value and the historical performance index value.

[0096] In implementation, the terminal can calculate the current performance weight factor for the previous target job type based on the current and historical performance index values ​​obtained for the previous target job type. In one example, the terminal can calculate the difference between the current and historical performance index values, calculate the proportion of this difference index value in the current performance index value, and use the calculated proportion as the current performance weight factor for the previous target job type.

[0097] For example, the terminal can calculate the current performance weight factor (denoted as p1) of the previous target job type using the following formula:

[0098] p1 = (z1 - z2) / z1,

[0099] Where z1 represents the current performance index value, z2 represents the historical performance index value, and (z1-z2) represents the difference between the current performance index value and the historical performance index value.

[0100] More specifically, the target performance event corresponding to the target job type can be memory usage, the current performance index value can be the memory usage of the previous target job type job, for example, 400M, and the historical performance index value can be the memory usage of the previous target job type job, for example, 300M. Then, the calculated current performance weight factor p1 of the previous target job type job is (400-300) / 400 = 0.25.

[0101] Step 204: Calculate the target weight based on the current performance weight factor, the weight factors of historical jobs, the preset weight, and the current weight value.

[0102] In practice, for a job of the previous target job type, the terminal can obtain the current performance weight factor, the weight factor of historical jobs, the preset weight, and the current weight value corresponding to the job of the previous target job type, calculate the result, and use the result as the target weight.

[0103] Step 206: Update the weights of the target job types in the target database to the target weights.

[0104] In practice, when the terminal determines that the calculated target weights meet the preset convergence conditions, it updates the weights corresponding to the target job types stored in the target database to the target weights calculated in the steps of the above embodiments.

[0105] In this embodiment, the weight value can be calculated based on the current performance index value and the historical performance index value to ensure the accuracy of the calculated weight value. Alternatively, the real-time performance of the calculated weight can be ensured by continuously updating the weights corresponding to different job types in the target database.

[0106] In one embodiment, such as Figure 3 As shown, the specific processing steps of step 204, "Calculate the target weight based on the current performance weight factor, the weight factors of historical jobs, the preset weight, and the current weight value," include:

[0107] Step 302: Calculate the weight factor difference between the current performance weight factor and the weight factors of historical jobs.

[0108] In practice, the weight factor difference represents the difference between different weight factors. The terminal can calculate the difference between the current performance weight factor and the weight factors of historical jobs. Based on this, the terminal can use the calculated difference between the current performance weight factor and the weight factors of historical jobs as the weight factor difference between the current performance weight factor and the weight factors of historical jobs.

[0109] Step 304: Calculate the weight adjustment factor based on the weight factor difference and the preset weight, and adjust the current weight value based on the weight adjustment factor to obtain the target weight.

[0110] In implementation, for a task of the previous target task type, the current weight value can be the weight value corresponding to the binding method of the previous target task type; the weight adjustment factor is a factor used to adjust the current weight value. The preset weight can be a pre-configured weight. Based on this, the terminal can calculate the weight adjustment factor based on the calculated weight factor difference and the preset weight, and adjust the current weight value of the task of the previous target task type based on the weight adjustment factor, and determine the adjusted weight value as the target weight.

[0111] In this embodiment, the current weight value is continuously adjusted by the weight factor difference degree, thereby achieving optimization of the target weight.

[0112] In one embodiment, such as Figure 4 As shown, the data processing method also includes:

[0113] Step 402: Collect the index values ​​corresponding to the target performance events of the job.

[0114] Among them, the target performance event corresponding to the job can be the performance event that has the most significant impact on the operation of the program corresponding to the job.

[0115] In practice, after obtaining a job, the terminal can determine the job type. In this way, the terminal can determine the target performance event corresponding to the job type and obtain the indicator value corresponding to the target performance event of the job.

[0116] Step 404: Calculate the first weight based on the index value corresponding to the target performance event of the job.

[0117] The metric values ​​corresponding to the target performance event for the job include the current performance metric value, historical performance metric values, current performance weight factor, historical performance weight factor, and current weight value for the job. The current performance metric value represents the metric value collected by the terminal from the metric corresponding to the target performance event after the job has run; the historical performance metric value represents the metric value collected by the terminal from the metric corresponding to the target performance event after the previous target job type job has run.

[0118] Similarly, the current performance weight factor represents the weight factor calculated based on the current performance index value and historical performance index value corresponding to the job, the historical performance weight factor represents the weight factor calculated based on the current performance index value and historical performance index value corresponding to the job of the previous target job type, and the current weight value represents the target weight corresponding to the job of the previous target job type.

[0119] In practice, the terminal can calculate the current performance weight factor based on the current performance index value and historical performance index value of the job, and perform calculations based on the calculated current performance weight factor, the weight factor of historical jobs, the preset weight and the current weight value, and use the calculated result as the first weight.

[0120] Step 406: If the first weight does not meet the preset convergence condition based on the target weight, update the weight value of the target weight of the target job type in the target database to the weight value of the first weight.

[0121] The preset convergence condition can be used to determine whether the calculated first weight has reached the optimal value compared to the target weight stored in the target database. For example, it could be to determine whether the calculated first weight is the same as the target weight stored in the target database, or whether the calculated first weight and the target weight stored in the target database fluctuate around the same weight value, etc.

[0122] In implementation, the terminal determines whether to update the target weights corresponding to the target job types stored in the target database, based on the calculated first weight and the target weights stored in the target database. If the terminal determines that the first weight does not meet the preset convergence condition, it will update the weight value corresponding to the target job type in the target database, which may be by updating the target weight value to the first weight value.

[0123] In one embodiment, the terminal may also update the weight value corresponding to the target job type in the target database based on the pre-configured correspondence between weight values ​​and binding methods.

[0124] In this embodiment, the current weight value is continuously adjusted by the weight factor difference degree, thereby achieving optimization of the target weight.

[0125] In one embodiment, the data processing method further includes:

[0126] If the target weight corresponding to the target job type is not found in the target database, the default weight is determined to be the target weight corresponding to the target job type, and the step of determining the target binding method corresponding to the target weight is executed based on the preset correspondence between weight and binding method.

[0127] In implementation, if the terminal determines that the target weight corresponding to the target job type is not found in the target database, the terminal can determine that this is the first time it has obtained a job of this target job type. Based on this, the terminal can determine the default weight as the target weight corresponding to this target job type, and determine the target binding method corresponding to this target weight based on the preset correspondence between weights and binding methods. In this way, the terminal can bind jobs of the target job type according to the target binding method. After binding a job of this target job type, the terminal can obtain the indicator value corresponding to the target performance event of this job (hereinafter referred to as the initial job), and calculate the current performance weight factor based on the obtained indicator value corresponding to the target performance event of the initial job; and calculate the target weight based on the current performance weight factor, the weight factor of historical jobs, the preset weight, and the current weight value.

[0128] In this embodiment, the comprehensiveness of job type statistics can be ensured by pre-configuring the target weights of the initial jobs.

[0129] In one embodiment, the data processing method further includes:

[0130] Add the mapping between target job types and target weights to the target database.

[0131] In practice, when the terminal determines the target job type as the initial job, the terminal can collect the indicator values ​​corresponding to the target performance events of the initial job after running the initial job, and calculate the target weight corresponding to the initial job based on the collected indicator values, thereby obtaining the correspondence between the target job type and the target weight of the initial job, that is, the correspondence between the target job type and the target weight corresponding to the target job type, and add the newly obtained correspondence between the target job type and the target weight to the target database.

[0132] In this embodiment, by continuously updating the correspondence between job types and target weights contained in the target database, it is possible to ensure the integration and classification of job types and their corresponding weight values, thus providing a foundation for subsequent applications.

[0133] In one embodiment, such as Figure 5 As shown, the specific processing procedure for step 402, "collecting the index values ​​corresponding to the target performance events of the job," includes:

[0134] Step 502: Create the performance event collection thread corresponding to the job.

[0135] Each job corresponds one-to-one with a performance event collection thread.

[0136] In implementation, the terminal responds to a job submission operation by obtaining the job corresponding to that operation. The terminal can create multiple job steps for the job through compute nodes and obtain the management PID for each job step. The management PID of each job step allows the terminal to obtain process information of applications related to the job. Based on this, the terminal can create a performance event collection thread corresponding to the job. This performance event collection thread can collect the metric values ​​of the target performance events of the job by running event collection functions.

[0137] Step 504: If it is determined that no recycling information has been received from the main thread, the main thread and target performance event corresponding to the job are determined through the performance event collection thread corresponding to the job, and the indicator value of the target performance event is collected.

[0138] In implementation, after creating the performance event collection thread corresponding to the job, the terminal can run the performance event collection thread. The terminal can run the event collection function in the performance event collection thread, which collects the performance event metric values ​​of the job. Specifically, after determining the main thread of the job, the terminal can determine whether the main thread has exited based on its exit flag. If no recycling information is received from the main thread, the terminal can determine that the main thread has not yet exited. In this way, the terminal can wait for the main thread's signal using a condition variable to delay, that is, collect performance events based on the main thread's performance event collection frequency. The terminal can obtain the application process list of the job, the application type corresponding to the application thread, and the target performance event corresponding to the job based on the management PID of the performance event collection thread. The terminal can initialize the target performance event, periodically read the metric value corresponding to the target performance event, and send the read metric value to a preset database until the job step ends. For example, if the terminal receives the recycling information sent by the job, it can determine that the job step and the main thread have ended, exit the loop, destroy the performance event collection thread corresponding to the job, and release the allocated variable space.

[0139] Optionally, the recycling information may be a thread recycling signal sent at the end of a job step, the target performance event may include software events and hardware events, and the default database may be an InfluxDB database.

[0140] In this embodiment, by creating an independent performance event collection thread for each job, the collection of performance event corresponding indicator values ​​between different jobs can be decoupled, ensuring the efficiency of performance event collection.

[0141] The following describes the specific execution process of the above data processing method in detail with reference to an embodiment:

[0142] SLURM, as a cluster workload manager, features high scalability and fault tolerance. This workload manager can allocate computing resources to users based on their needs; it also provides a framework for monitoring jobs running on compute nodes; and it can arbitrate computing resources among multiple running jobs. In other words, the SLURM system can allocate various computing resources, including CPU, memory, and GPU resources, and can monitor the consumption of these resources. However, in related technologies, SLURM cannot assess and optimize performance bottlenecks in job execution. The SLURM scheduling system cannot perform inter-core binding based on job characteristics, and mismatched binding methods can lead to inefficient job execution.

[0143] Based on this, this embodiment provides a data processing method for implementing job performance event collection and performance optimization strategies based on the SLURM system. Specifically, when a job is submitted, SLURM first compares the information of the same job type stored in the target database and determines the CPU binding method or GPU binding method based on the job type. During job execution, SLURM can also collect the application type of the job and the count (i.e., metric value) of the corresponding performance events and store it in the database. It updates the CPU and GPU binding methods based on the job's performance event counts to optimize job execution efficiency. It also allows SLURM users and developers to view different job types and job performance event counts in real time, identify job performance bottlenecks, and provides an interface for manually modifying job binding strategies. The terminal can obtain the binding strategy manually input by the user through this job binding strategy interface, determine the binding method corresponding to the binding strategy, and bind and process the job based on the binding method.

[0144] like Figure 6a The diagram illustrates a process for adjusting the binding method of jobs within compute nodes for jobs of the same type. The terminal can determine the optimal binding method for different application types based on the count of target performance events (target performance event metrics). After submitting a job, binding is performed according to the trained binding rules, while job-related records are recorded in the database. The binding rules are then corrected, and appropriate inter-core bindings are set for queued jobs based on historical job data.

[0145] Specifically, after a job is submitted, jobs of the same type are retrieved from the database. If a job of the same type is found in the database, the historical performance event weight x can be obtained from the database. That is, the historical performance event weight x corresponding to this job type stored in the database is used as the target weight for this job type. An appropriate binding method is then selected based on the weight value x. In other words, based on the preset correspondence between binding methods and weights, the target binding method corresponding to the weight value x is determined. In this way, the job can be bound and run based on this target binding method, and performance event collection can begin for that job. When the job exits, the weight value x is calculated based on the process time and updated in the database.

[0146] If no job of the same type is found in the database, a default time weight value x is set, and an appropriate binding method is selected based on the weight value x. That is, based on the preset correspondence between binding methods and weights, the target binding method corresponding to the weight value x is determined. In this way, the job can be bound and run based on the target binding method, and performance event collection can begin for the job. When the job step exits, the weight value x is calculated based on the process time and updated in the database.

[0147] Since the count value of performance events reflects the current job execution efficiency, the terminal can select different binding methods based on the count value of performance events to optimize job execution efficiency.

[0148] For example, for the current task, the target weight x' of the next task of the same type can be calculated using the following formula:

[0149] x' = x1 + k1(p1 - p2),

[0150] Where x1 represents the target weight corresponding to the current job, p1 represents the current performance weight factor of the current job, p2 represents the weight factor of the previous job of the same type, i.e. the weight factor of historical jobs; k1 represents the preset weight, for example, k1 = 1 / 10, which is the target weight of the next job of the same type calculated by x'.

[0151] The pre-defined correspondence between weights and binding methods can be shown in Table 1 below:

[0152] Table 1

[0153]

[0154] The process of adjusting the correspondence between job types and target weights in the target database by the terminal can also include: if the terminal determines that the calculated weight value x' has reached its maximum, it sets it to the minimum value in Table 1; if the weight value x' reaches its minimum, it adjusts it to the maximum value in Table 1. During the continuous iteration of the job, the weight value will fluctuate within a range to achieve the ideal binding effect.

[0155] The terminal can create an independent performance event collection thread for each job, such as Figure 6b The diagram illustrates the performance event collection process for a job. In the SLURM system, the terminal can initialize job step-related structures. Specifically, it creates multiple job steps and initializes these structures after creation. The terminal can also obtain the job step management PID, which is used to retrieve the job application process PID. The terminal obtains the management PID within the job step's process, and through this PID, it can acquire process information related to the job's applications. The terminal can create a separate performance event collection thread for the job to run the event collection function. This new thread contains a while loop, the duration of which can be controlled by a non-blocking condition variable. If the main thread does not exit, the new thread first retrieves the job's application process list based on the management PID, then initializes performance events such as software and hardware events, and periodically reads performance event data and sends it to the InfluxDB database until the job step ends.

[0156] Specifically, the event acquisition function can run as follows: It checks the main thread's exit flag and waits for a delay using a condition variable. If the flag is set, it retrieves the process list and application type within the job application process, initializes the required monitoring events (including software and hardware events), and periodically reads and sends the collected data to the database for both software and hardware events (e.g., via HTTP). After the job step ends and the sending thread reclaims the signal, it checks if the flag is set to "no." If so, the event acquisition function exits the while loop, destroys the thread, releases the allocated variable space, and terminates the event acquisition function's execution, thus ending the performance event acquisition thread's execution.

[0157] Performance events are used to assess bottlenecks during job execution and to formulate job binding strategies. Currently collected metrics include, but are not limited to, the following performance events: CPU-related metrics include: CPU utilization, CPU frequency, IPC (instructions), SIMD, L3 hit events, L2 hit events, branchless prediction, page misses, CPU context migration, etc.; memory-related metrics include: memory utilization, memory read / write bandwidth, etc.; storage-related metrics include: IOPS, etc.; network-related metrics include: Eth.bw; GPU and DCU-related metrics include: VRAM%, GPU%, etc.

[0158] The terminal collects different performance events for different job types. The classification method is to determine whether the jobs belong to the same user and are of the same type. The terminal can also categorize jobs of the same type from the same user. Since jobs of the same type include CPU version and GPU version, as shown in Table 2 below, the selection of performance event monitoring is based on CPU version, GPU version, and multiple nodes.

[0159] Table 2

[0160]

[0161] This embodiment provides a data processing method that enables real-time performance event monitoring during job execution within a SLURM scheduling system. It compares the sum of the current performance event count weights with existing values ​​in the database to determine whether the current job's performance events have increased or decreased compared to previous jobs, thereby adjusting the binding method for subsequent jobs of the same type. The optimal binding method is achieved based on the job performance event count and the job application type. As the number of jobs increases, the binding method gradually approaches or fluctuates around the ideal value. In other words, it allows the SLURM scheduling system to collect job performance events in real time, such as cache misses, CPU migrations, and branch predictions. By integrating the SLURM scheduling system with performance event collection, and intelligently selecting the binding method based on the job type after the job is submitted to the SLURM scheduling system, it improves job execution efficiency.

[0162] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0163] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0164] In one embodiment, such as Figure 7 As shown, a data processing device 700 is provided, including: a first acquisition module 702, a query module 704, and a first determination module 706, wherein:

[0165] The first acquisition module 702 is used to acquire jobs and determine the target job type of the job.

[0166] The query module 704 is used to query the target weight corresponding to the target job type in the target database. The target weight is calculated based on the indicator value corresponding to the target performance event of the previous target job type.

[0167] The first determining module 706 is used to determine the target binding method corresponding to the target weight based on the preset correspondence between weight and binding method, and bind the job based on the target binding method.

[0168] In one embodiment, the metric value corresponding to the target performance event of the previous target job type includes the current performance metric value, historical performance metric value, current performance weight factor, historical performance weight factor, and current weight value. The device further includes:

[0169] The first calculation module is used to calculate the current performance weighting factor based on the current performance index value and the historical performance index value.

[0170] The second calculation module is used to calculate the target weight based on the current performance weight factor, the weight factors of historical jobs, the preset weight, and the current weight value.

[0171] The first update module is used to update the weights of the target job types in the target database to the target weights.

[0172] In one embodiment, the second computing module is specifically used for:

[0173] Calculate the weight factor difference between the current performance weight factor and the weight factors of historical jobs;

[0174] The weight adjustment factor is calculated based on the weight factor difference and the preset weight, and the current weight value is adjusted based on the weight adjustment factor to obtain the target weight.

[0175] In one embodiment, the data processing apparatus further includes:

[0176] The data acquisition module is used to collect the metric values ​​corresponding to the target performance events of the job.

[0177] The third calculation module calculates the first weight based on the index value corresponding to the target performance event of the job;

[0178] The second update module is used to update the weight value of the target weight of the target job type in the target database to the weight value of the first weight when the first weight determined based on the target weight does not meet the preset convergence condition.

[0179] In one embodiment, the data processing apparatus further includes:

[0180] The second determining module is used to determine the default weight as the target weight corresponding to the target job type if the target weight corresponding to the target job type is not found in the target database, and to perform the step of determining the target binding method corresponding to the target weight based on the preset correspondence between weight and binding method.

[0181] In one embodiment, the data processing apparatus further includes:

[0182] Add a module to add the mapping between target job types and target weights to the target database.

[0183] In one embodiment, the acquisition module is specifically used for:

[0184] Create a performance event collection thread corresponding to the job; if it is determined that no recycling information has been received from the main thread, use the performance event collection thread corresponding to the job to determine the main thread and target performance event of the job, and collect the metric value of the target performance event.

[0185] Each module in the aforementioned data processing device 700 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0186] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores job-related data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0187] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0188] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0190] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0191] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0192] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0193] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0194] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the job and determine the target job type of the job; The target weight corresponding to the target job type is queried in the target database. The target weight is calculated based on the indicator value corresponding to the target performance event of the previous target job type. The indicator value corresponding to the target performance event of the previous target job type includes the current performance indicator value, historical performance indicator value, current performance weight factor, historical performance weight factor, and current weight value. Based on the preset correspondence between weights and binding methods, the target binding method corresponding to the target weight is determined, and the job is bound based on the target binding method; The method further includes: Calculate the current performance weighting factor based on the current performance index value and the historical performance index value; Calculate the weight factor difference between the current performance weight factor and the weight factors of historical jobs; calculate the weight adjustment factor based on the weight factor difference and the preset weight, and adjust the current weight value based on the weight adjustment factor to obtain the target weight; Update the weights of the target job type in the target database to the target weights.

2. The method according to claim 1, characterized in that, Following the step of binding the job based on the target binding method, the method further includes: Collect the index values ​​corresponding to the target performance events of the job; Calculate the first weight based on the index value corresponding to the target performance event of the task; If, based on the target weight, it is determined that the first weight does not meet the preset convergence condition, the weight value of the target weight of the target job type in the target database is updated to the weight value of the first weight.

3. The method according to claim 1, characterized in that, Also includes: If the target weight corresponding to the target job type is not found in the target database, the default weight is determined to be the target weight corresponding to the target job type, and the step of determining the target binding method corresponding to the target weight based on the preset correspondence between weight and binding method is executed.

4. The method according to claim 3, characterized in that, Also includes: Add the correspondence between the target job type and the target weight to the target database.

5. The method according to claim 2, characterized in that, The collection of metric values ​​corresponding to the target performance events of the job includes: Create a performance event collection thread corresponding to the job; If it is determined that no recycling information has been received from the main thread, the main thread and target performance event corresponding to the job are determined through the performance event collection thread corresponding to the job, and the indicator value of the target performance event is collected.

6. The method according to claim 1, characterized in that, The job acquisition includes: In response to a job submission operation, obtain the job corresponding to the job submission operation; or... After receiving the job submission notification message, obtain the job corresponding to the job submission notification message.

7. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire jobs and determine the target job type of the jobs; The query module is used to query the target weight corresponding to the target job type in the target database. The target weight is calculated based on the indicator value corresponding to the target performance event of the previous target job type. The indicator value corresponding to the target performance event of the previous target job type includes the current performance indicator value, historical performance indicator value, current performance weight factor, historical performance weight factor, and current weight value. The first determining module is used to determine the target binding method corresponding to the target weight based on a preset correspondence between weights and binding methods, and to bind the job based on the target binding method; The device further includes: The first calculation module is used to calculate the current performance weighting factor based on the current performance index value and the historical performance index value. The second calculation module is used to calculate the weight factor difference between the current performance weight factor and the weight factors of historical jobs; calculate the weight adjustment factor based on the weight factor difference and the preset weight; and adjust the current weight value based on the weight adjustment factor to obtain the target weight. The first update module is used to update the weight of the target job type in the target database to the target weight.

8. The apparatus according to claim 7, characterized in that, Also includes: The second determining module is used to determine the default weight as the target weight corresponding to the target job type if the target weight corresponding to the target job type is not found in the target database, and to execute the step of determining the target binding method corresponding to the target weight based on the preset correspondence between weight and binding method.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images