Artificial intelligence-based scheduling optimization method and device, computer device and medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a scheduling optimization method and device based on artificial intelligence, computer equipment and a medium. The method processes the job content and the amount of computing resources of a to-be-optimized job through a time length prediction model to obtain a predicted job time length, obtains an upstream job corresponding to a downstream job according to a similarity calculation result of the upstream job and the downstream job in the to-be-optimized job, determines a starting time point of the corresponding downstream job according to a starting time point of the upstream job and the predicted job time length, inputs the starting time point of the downstream job, the predicted job time length and the amount of computing resources into a scheduling optimization model to obtain an optimization amount of the starting time point and the amount of computing resources of the downstream job, avoids the influence of job content change on the optimization scheduling accuracy, adjusts the scheduling time and the amount of computing resources of the downstream job according to the overall job condition, can smooth the peak value of the scheduling resource amount, reasonably allocates the computing resources and improves the efficiency of job execution.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a scheduling optimization method, apparatus, computer equipment, and medium based on artificial intelligence. Background Technology

[0002] Currently, with the improvement of computer computing performance, data processing jobs are usually adopted in the form of computer jobs to improve the efficiency of data processing. The job content includes data retrieval, data transformation, data calculation, etc. Existing jobs are usually divided into upstream jobs and downstream jobs according to the data relationship. When the upstream job is completed, the corresponding downstream job is scheduled.

[0003] However, in real-world scenarios, computing resources are limited. Using the preset completion time of upstream jobs as the trigger for scheduling downstream jobs can lead to problems. Since upstream jobs may complete too early or too late, triggering based on the preset completion time can cause downstream jobs to be scheduled too early or too late, before sufficient computing resources are available. This can result in peak resource usage exceeding limits, leading to low efficiency for downstream job execution. Therefore, how to rationally schedule downstream jobs to improve their efficiency is a pressing issue. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a scheduling optimization method, apparatus, computer equipment and medium based on artificial intelligence to solve the problems of unreasonable downstream job scheduling and low job execution efficiency.

[0005] In a first aspect, embodiments of the present invention provide a scheduling optimization method based on artificial intelligence, the scheduling optimization method comprising:

[0006] Obtain the job content and pre-allocated computing resources for each upstream and downstream job in the job to be optimized. Input the job content and the pre-allocated computing resources into the trained duration prediction model to obtain the predicted job duration.

[0007] For any downstream job in the job to be optimized, the similarity between the job content of each upstream job in the job to be optimized and the job content of the downstream job is calculated, and the target upstream job corresponding to the downstream job is obtained based on the comparison between the similarity calculation result and the preset threshold.

[0008] Determine the start time of the corresponding downstream operation based on the start time and predicted operation duration of each upstream operation.

[0009] The start time, predicted job duration, and pre-allocated computing resources of each downstream job are input into the pre-trained scheduling optimization model to obtain the optimized start time and optimized computing resources of the corresponding downstream job. This is used to instruct the corresponding downstream job to be scheduled to the optimized computing resources at the optimized start time.

[0010] Secondly, embodiments of the present invention provide a scheduling optimization device based on artificial intelligence, the scheduling optimization device comprising:

[0011] The duration prediction module is used to obtain the job content and pre-allocated computing resources of each upstream job and each downstream job in the job to be optimized, and input the job content and the pre-allocated computing resources into the trained duration prediction model to obtain the predicted job duration.

[0012] The similarity calculation module is used to calculate the similarity between the content of each upstream job in the job to be optimized and the content of the downstream job for any downstream job in the job to be optimized, and to obtain the target upstream job corresponding to the downstream job based on the comparison result of the similarity calculation result and the preset threshold.

[0013] The timing determination module is used to determine the start time of the corresponding downstream operation based on the start time of the upstream operation of each target and the predicted operation duration.

[0014] The resource scheduling module is used to input the start time, predicted job duration, and pre-allocated computing resources of each downstream job into the pre-trained scheduling optimization model to obtain the optimized start time and optimized computing resources of the corresponding downstream job. This module is used to instruct the corresponding downstream job to be scheduled to the optimized computing resources at the optimized start time.

[0015] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the scheduling optimization method as described in the first aspect.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the scheduling optimization method as described in the first aspect.

[0017] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0018] Obtain the job content and pre-allocated computing resources for each upstream and downstream job in the job to be optimized. Input the job content and pre-allocated computing resources into a pre-trained duration prediction model to obtain the predicted job duration. For any downstream job in the job to be optimized, calculate the similarity between the job content of each upstream job and the job content of the downstream job. Based on the comparison between the similarity calculation result and a preset threshold, obtain the target upstream job corresponding to the downstream job. Based on the start time point and predicted job duration of each target upstream job, determine the start time point of the corresponding downstream job. Input the start time point, predicted job duration, and pre-allocated computing resources of each downstream job into a pre-trained model. The scheduling optimization model obtains the optimized start time and optimized computing resource amount for the corresponding downstream jobs. This is used to instruct the corresponding downstream jobs to be scheduled to the optimized computing resource amount at the optimized start time. The job content of the upstream jobs is input into the duration prediction model to obtain the predicted duration. Then, the initial time point of the corresponding downstream jobs is determined based on the predicted duration of the upstream jobs. This avoids the impact of job content changes on the accuracy of the optimized scheduling, thereby affecting the efficiency of job execution. Furthermore, by using the optimized scheduling model to optimize and adjust the initial time point and initial resource amount, the scheduling time and computing resource amount of downstream jobs can be adjusted according to the overall job situation, smoothing the peak of scheduling resource amount, thereby rationally allocating computing resources and improving the efficiency of job execution. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an application environment for a scheduling optimization method based on artificial intelligence provided in Embodiment 1 of the present invention;

[0021] Figure 2 This is a flowchart illustrating a scheduling optimization method based on artificial intelligence provided in Embodiment 1 of the present invention;

[0022] Figure 3 This is a flowchart illustrating a scheduling optimization method based on artificial intelligence provided in Embodiment 2 of the present invention;

[0023] Figure 4 This is a schematic diagram of the structure of a scheduling optimization device based on artificial intelligence provided in Embodiment 3 of the present invention;

[0024] Figure 5This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0026] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0029] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0032] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0033] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0034] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0035] The scheduling optimization method based on artificial intelligence provided in Embodiment 1 of this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0036] See Figure 2 This is a flowchart illustrating a scheduling optimization method based on artificial intelligence provided in Embodiment 1 of the present invention. The above-described scheduling optimization method can be applied to... Figure 1 In this context, the server-side computer connects to the client, which acts as the execution entity for the job, to access the client and obtain information about the job to be optimized. This information includes the job content and the amount of pre-allocated computing resources. The server is equipped with a pre-trained duration prediction model and a pre-trained scheduling optimization model, used to optimize job scheduling based on the information about the job to be optimized. Figure 2 As shown, the scheduling optimization method may include the following steps:

[0037] Step S201: Obtain the job content and pre-allocated computing resources for each upstream and downstream job in the job to be optimized. Input the job content and pre-allocated computing resources into the trained duration prediction model to obtain the predicted job duration.

[0038] Here, "job" can refer to computational jobs related to data processing, and "data processing" can refer to operations that analyze and process data, such as data retrieval, data transformation, data sorting, and data calculation. "Computational job" can refer to a transaction process that uses a computer to perform computational content, and "job to be optimized" can refer to a job that requires optimization of computing resource scheduling.

[0039] The jobs to be optimized include upstream jobs and downstream jobs. Upstream jobs and downstream jobs are divided according to the relationship between data. Upstream jobs can be jobs that do not require data from other jobs, while downstream jobs can be jobs that require data from other jobs.

[0040] The job content can refer to the specific calculation content of the job, and the amount of computing resources can refer to the CPU usage, memory allocation space, and network bandwidth, etc. The implementer can pre-allocate the amount of computing resources for each job to be optimized according to the actual situation.

[0041] In this embodiment, the duration prediction model can refer to a convolutional neural network, which is used to predict the duration of the corresponding job to be optimized based on the job content and the amount of pre-allocated computing resources. The duration prediction model extracts features from the semantic information of the job content to obtain semantic feature information such as the amount of work and the difficulty of the job content. The semantic feature information is combined with the amount of work per unit time corresponding to the amount of pre-allocated computing resources to obtain the predicted job duration.

[0042] In one implementation, the duration prediction model can also be a logistic regression model, a fully convolutional neural network model, etc.

[0043] The above steps involve obtaining the job content and pre-allocated computing resources for each upstream and downstream job in the job to be optimized, inputting the job content and pre-allocated computing resources into a trained duration prediction model to obtain the predicted job duration. Based on the job content and pre-allocated computing resources of the job to be optimized, the predicted job duration of the job to be optimized is obtained through the duration prediction model. This can avoid the situation where the predicted job duration is inaccurate when the job content of the job to be optimized changes, thereby improving the accuracy of subsequent scheduling optimization.

[0044] Step S202: For any downstream job in the job to be optimized, calculate the similarity between the job content of each upstream job in the job to be optimized and the job content of the downstream job, and obtain the target upstream job corresponding to the downstream job based on the comparison between the similarity calculation result and the preset threshold.

[0045] In this embodiment, similarity can refer to cosine similarity, and the formula for calculating cosine similarity is: Where Sim is the cosine similarity, A is the word vector of the upstream task a, B is the word vector of the downstream task b, |A| is the modulus of the word vector of the upstream task a, and |B| is the modulus of the word vector of the downstream task b.

[0046] A preset threshold can refer to a threshold used to determine whether the content of a corresponding upstream operation is related to the content of a corresponding downstream operation. A target upstream operation can refer to an upstream operation that is related to the downstream operation.

[0047] In one implementation, similarity can also refer to Euclidean distance, edit distance, etc.

[0048] Specifically, if the similarity calculation result between the content of the corresponding upstream operation and the content of the corresponding downstream operation is greater than or equal to a preset threshold, then the corresponding upstream operation will be used as the target upstream operation of the corresponding downstream operation. If the similarity calculation result between the content of the corresponding upstream operation and the content of the corresponding downstream operation is less than the preset threshold, then the corresponding upstream operation will not be used as the target upstream operation of the corresponding downstream operation.

[0049] In this embodiment, the preset threshold is set to 0 to ensure that upstream jobs associated with downstream jobs are not missed. The implementer can adjust the preset threshold according to the actual situation. For example, if the implementer only needs a rough job scheduling optimization, the preset threshold can be increased, for example, to 0.2, thereby saving the optimization time of the scheduling optimization model. It should be noted that since the value range of cosine similarity is [0, 1], it is recommended that the value range of the preset threshold also be [0, 1].

[0050] Optionally, for any downstream job in the job to be optimized, the similarity calculation between the job content of each upstream job and the job content of each downstream job in the job to be optimized includes:

[0051] The keyword vector is obtained by searching for keywords in the assignment content using a pre-defined dictionary model.

[0052] The cosine similarity between the keyword vectors of the corresponding upstream task and the keyword vectors of the corresponding downstream task is determined as the similarity calculation result.

[0053] Among them, the dictionary model can refer to the retrieval model composed of word vectors corresponding to the words contained in the historical assignment content. The dictionary model includes the keyword dictionary model.

[0054] Specifically, the upstream and downstream tasks in the task to be optimized are segmented into words using a dictionary model. Then, the words obtained after segmentation are filtered using a keyword dictionary model to obtain keywords. The keyword sequence is converted into keyword vectors using a pre-trained word vector conversion model, which can refer to the Word2Vec model, neural network language model, recurrent neural network language model, etc.

[0055] In this embodiment, the keyword vectors of the task content are compared, thereby isolating irrelevant information and making the extracted similarity more closely related to the key information of the task content, thus improving the accuracy of subsequent selection of target upstream tasks based on similarity for corresponding downstream tasks.

[0056] For any downstream task in the task to be optimized, the similarity between the content of each upstream task and the content of the downstream task is calculated. Based on the comparison between the similarity calculation result and a preset threshold, the target upstream task step corresponding to the downstream task is obtained. By comparing the similarity and threshold results, the upstream and downstream tasks in the task to be optimized are associated, which can more accurately determine the start time of the downstream task, improve the accuracy of the input of the scheduling optimization model, and thus improve the accuracy of the scheduling optimization model.

[0057] Step S203: Determine the start time of the corresponding downstream operation based on the start time of each upstream operation and the predicted operation duration.

[0058] The start time point can refer to the time when the operation begins to be executed. The start time point of the upstream operation is usually set in advance by the implementer.

[0059] Specifically, by adding the start time of the upstream operation to the predicted operation duration, the end time of the upstream operation can be obtained. Each target upstream operation corresponds to an end time.

[0060] Since the execution of downstream operations requires data from the corresponding upstream operations, the start time of the downstream operation must be after the end time of the corresponding upstream operation. The maximum value of the end times of all the upstream operations is taken as the start time of the downstream operation. The start time of the downstream operation means the earliest time when the downstream operation starts to be executed.

[0061] For example, the target upstream operations corresponding to downstream operation c are upstream operation a and upstream operation b. The start time of upstream operation a is the preset 10:00, the predicted operation duration of upstream operation a is 30 minutes, and the end time of upstream operation a is 10:30. The start time of upstream operation b is the preset 8:00, the predicted operation duration of upstream operation b is 300 minutes, and the end time of upstream operation b is 13:00. Therefore, the start time of downstream operation c is 13:00.

[0062] The above steps, which determine the start time of the corresponding downstream task based on the start time of each upstream task and the predicted task duration, not only determine the start time of the downstream task, but also provide constraints for the downstream task during the optimization process. That is, the execution time of the downstream task must be after its start time, thereby avoiding the situation where the execution time of the downstream task is too early and it still needs to wait after the computing resources are allocated, thus improving the utilization efficiency of computing resources.

[0063] Step S204: Input the start time, predicted job duration, and pre-allocated computing resources of each downstream job into the pre-trained scheduling optimization model to obtain the optimized start time and optimized computing resources of the corresponding downstream job.

[0064] In this embodiment, the scheduling optimization model adopts an autoencoder model.

[0065] The optimized start time and optimized computing resources for the corresponding downstream job are used to instruct the corresponding downstream job to be scheduled to the optimized computing resources at the optimized start time.

[0066] In one implementation, the scheduling optimization model may also employ a convolutional neural network model, a generative adversarial network model, or the like.

[0067] Specifically, for any downstream job, the start time, predicted job duration, and pre-allocated computing resources of the corresponding downstream job are concatenated into a one-dimensional vector. The input to the scheduling optimization model consists of a one-dimensional vector corresponding to each downstream job, with each vector corresponding to an input neuron.

[0068] The output of the scheduling optimization model consists of multiple one-dimensional vectors, with the number of output vectors matching the number of input vectors. Each output vector corresponds to a downstream job, representing the optimized start time and optimized computational resource amount for that downstream job.

[0069] Optionally, the scheduling optimization model includes a feature encoder and a feature decoder. The sample time point, sample job duration and sample resource quantity corresponding to each historical downstream job are used as the pre-training samples of the scheduling optimization model, the actual time point and actual resource quantity corresponding to each historical downstream job are used as the pre-training labels of the scheduling optimization model, and Euclidean distance is used as the pre-training loss function.

[0070] The pre-training process of the scheduling optimization model includes:

[0071] The pre-trained samples are input into the feature encoder for feature extraction to obtain the sample feature tensor;

[0072] Input the sample feature tensor into the feature decoder to obtain the predicted time point and predicted resource quantity for each historical downstream operation;

[0073] The pre-training loss function is calculated based on the predicted time point, predicted resource quantity, and pre-training labels. The parameters of the feature encoder and feature decoder are updated through backpropagation based on the calculation results until the pre-training loss function converges, thus obtaining the pre-trained scheduling optimization model.

[0074] Here, the sample time point, sample job duration, and sample resource quantity can refer to the preset start time point, job duration, and pre-allocated computing resources when the historical downstream job has not been optimized. The actual time point and actual resource quantity can refer to the optimized start time point and optimized computing resources of the historical downstream job.

[0075] Specifically, the output of the feature decoder is a K-row, two-column matrix, where the elements in the first column represent the predicted time points and the elements in the second column represent the predicted resource quantities. The pre-trained labels are also a K-row, two-column matrix, where the elements in the first column represent the actual time points and the elements in the second column represent the actual resource quantities.

[0076] The formula for calculating Euclidean distance is: Where L1 represents the Euclidean distance, x ik Let x′ represent the element in the k-th row and i-th column of the output matrix of the feature decoder. i This represents the element in the k-th row and i-th column of the matrix corresponding to the pre-trained labels.

[0077] This embodiment uses an optimized scheduling model to adjust the scheduling time and computing resources of downstream operations based on the overall operation situation. It can extract the potential correlation between downstream operations and thus allocate computing resources reasonably.

[0078] Optionally, after inputting the sample feature tensor into the feature decoder and outputting the prediction time point and prediction resource amount, the following is also included:

[0079] For any historical downstream operation, calculate the ratio of the product of the sample operation duration and the sample resource quantity of the corresponding historical downstream operation to the predicted resource quantity of the corresponding historical downstream operation, and determine the ratio as the sample predicted duration of the corresponding historical downstream operation.

[0080] Based on the predicted time points of the corresponding historical downstream operations and the sample predicted duration of the corresponding historical downstream operations, the predicted time period of the corresponding historical downstream operations is determined.

[0081] For any target time point, if the prediction time period of the corresponding historical downstream operation includes the target time point, then the corresponding historical downstream operation is retained, and the sum of the predicted resource amounts of all retained historical downstream operations is determined to be the total predicted resource amount for the corresponding target time point.

[0082] The maximum value in the total predicted resources for each target time point is determined as the peak loss term, and the peak loss term is updated to the pre-trained loss function by weighted summation.

[0083] Here, the sample prediction duration can refer to the prediction duration required for a historical downstream job to execute when the allocated computing resources are equal to the predicted resources. The prediction time period can refer to the time period during which the prediction of a historical downstream job was executed. The total predicted resources can refer to the sum of the predicted resources for all downstream jobs at the target time point.

[0084] The peak loss term can refer to the loss function term corresponding to the peak constraint of the total predicted resources.

[0085] Weighted summation can refer to multiplying the peak loss term by its corresponding weight and then adding it to the pre-trained loss function.

[0086] Specifically, the peak loss term is denoted by L2, L2 = max(S t ), where S t Let be the total predicted resources at the t-th target time point, where t ranges from [1, 2, ..., T], T is the total number of target time points, and max indicates taking the maximum value.

[0087] The larger the peak value of the predicted total resources, the larger the peak loss term. During the training process, the scheduling optimization model will adjust the parameters to make the peak value of the predicted total resources as small as possible, so as to more effectively smooth the amount of computational resources at each time point after optimization.

[0088] In this embodiment, the resource allocation is constrained by the peak loss term, which provides a learning direction for training. As a result, when scheduling the job to be optimized according to the optimized start time point and the optimized computing resources output by the trained scheduling optimization model, the total predicted resources at each target time point are relatively small, which plays a role in smoothing the prediction of the total resources and improving the execution efficiency of the job.

[0089] Optionally, after obtaining the predicted time period for each historical downstream operation, the following is also included:

[0090] For any historical downstream operation, determine the predicted termination point of the corresponding historical downstream operation based on the predicted time period of the corresponding historical downstream operation.

[0091] The maximum value of the predicted termination point for each historical downstream job is determined as the time loss term, and the time loss term is updated to the pre-trained loss function by weighted summation.

[0092] Here, the prediction termination point can refer to the end time point of the prediction period, and the time loss term can refer to the loss function term corresponding to the task completion time constraint.

[0093] Specifically, the time loss term is denoted by L3, L3 = max(Q k ), where Q k Let K be the predicted termination point corresponding to the kth historical downstream operation. The value of k ranges from [1, 2, ..., K], and K is the total number of historical downstream operations.

[0094] The larger the maximum value of the predicted termination point, the greater the time loss term. During the training process, the scheduling optimization model will adjust the parameters to minimize the maximum value of the predicted termination point, thereby improving the efficiency of the job to be optimized.

[0095] In this embodiment, the resource allocation is constrained by the peak loss term, which provides a learning direction for training. As a result, when scheduling the jobs to be optimized according to the optimized start time and optimized computing resources output by the trained scheduling optimization model, the completion time of all jobs to be optimized is smaller, thus improving the execution efficiency of the jobs.

[0096] Optionally, the peak loss term corresponds to the peak weight, and the time loss term corresponds to the time weight;

[0097] Before calculating the pre-training loss function based on the predicted time point, predicted resource quantity, and pre-training labels, the following steps are also included:

[0098] If the peak loss term is greater than the preset resource threshold, the peak weight is increased, and the corresponding peak weight parameter in the pre-trained loss function is updated with the increased peak weight.

[0099] If the time loss term exceeds the preset time threshold, the time weight is increased, and the corresponding time weight parameter in the pre-training loss function is updated with the increased time weight.

[0100] Peak weight can refer to the degree of influence of the peak loss term on the gradient during training, while time weight can refer to the degree of influence of the time loss term on the gradient during training.

[0101] Resource threshold can refer to the maximum total amount of computing resources at any pre-set time point, while time threshold can refer to the latest completion time of the job to be optimized, which is set in advance.

[0102] Specifically, the initial values ​​of time weight and peak weight are both set to the same value. When updating peak weight or time weight, the value of each update is a fixed value. In this embodiment, the same value is set to 1, and the fixed value is set to 0.5.

[0103] For example, after introducing peak loss and time loss, the pre-training loss function is expressed as: L=L1+w1L2+w2L3, where w1 is the peak weight parameter and w2 is the time weight parameter. Initially, w1=w2=1. If the peak loss term is greater than the preset resource threshold, then w1 is increased to 1.5.

[0104] This embodiment dynamically adjusts the corresponding weights according to preset constraints, providing a more accurate direction for the training of the scheduling optimization model, improving training efficiency, and ensuring that the optimized scheduling meets the preset conditions.

[0105] The above steps involve inputting the start time, predicted job duration, and pre-allocated computing resources of each downstream job into a pre-trained scheduling optimization model to obtain the optimized start time and optimized computing resources for the corresponding downstream job. By optimizing the start time and computing resources of each downstream job through the scheduling optimization model, the peak scheduling resource volume can be smoothed, thereby improving the efficiency of job execution.

[0106] This embodiment inputs the content of upstream operations into the duration prediction model to obtain the predicted duration. Then, based on the predicted duration of upstream operations, it determines the initial time point of the corresponding downstream operations. This avoids changes in operation content from affecting the accuracy of optimized scheduling, thereby impacting the efficiency of operation execution. Furthermore, by using an optimized scheduling model to optimize and adjust the initial time point and initial resource quantity, the scheduling time and computing resource quantity of downstream operations can be adjusted according to the overall operation situation, smoothing the peak of scheduling resource quantity, thereby rationally allocating computing resources and improving the efficiency of operation execution.

[0107] See Figure 3 This is a flowchart illustrating a scheduling optimization method based on artificial intelligence provided in Embodiment 2 of the present invention. In this scheduling optimization method, the duration prediction model adopts a convolutional neural network, and the trained duration prediction model includes a trained semantic encoder and a trained fully connected layer.

[0108] The training process of the above duration prediction model includes:

[0109] Step S301: Input the task content into the trained semantic encoder for feature extraction to obtain the semantic feature tensor.

[0110] Step S302: Concatenate the embedding vector obtained by mapping the computational resource quantity with the semantic feature tensor to obtain the concatenated feature vector.

[0111] Step S303: Input the concatenated feature vector into the trained fully connected layer to obtain the predicted job duration.

[0112] The semantic encoder includes convolutional layers and pooling layers. Convolutional layers are used to extract features, pooling layers are used for dimensionality reduction, and fully connected layers are used to map feature tensors to the output space.

[0113] Embedding vectors can refer to the vector form of computational resources, so that computational resources can be fused with semantic feature tensors as feature information.

[0114] Specifically, the size of the job content to be optimized needs to be adjusted before it is input into the trained semantic encoder. The maximum size of the job content of the historical job is used as the reference size, and the size of the job content to be optimized is expanded to the reference size by padding with zero characters.

[0115] The embedding vector and the semantic feature tensor can be concatenated by using a concatenation method to obtain the concatenated feature tensor.

[0116] Before inputting the trained fully connected layer, the concatenated feature tensor is flattened to reduce the data dimensionality of the concatenated feature tensor.

[0117] In one implementation, the embedding vector and the semantic feature tensor can also be concatenated by adding the embedding vector to the semantic feature tensor position by position.

[0118] This embodiment uses embedded vectors to fuse the computational resource quantity with the feature tensor of the job content to be optimized, thereby improving the effectiveness of the features and thus improving the accuracy of the prediction of the duration of the job to be optimized.

[0119] Corresponding to the AI-based scheduling optimization method in the above embodiments, Figure 4 A structural block diagram of an artificial intelligence-based scheduling optimization device provided in Embodiment 3 of the present invention is shown. This scheduling optimization device is applied to a server. The computer device corresponding to the server is connected to a client, which acts as the main body for job execution, to access the client and obtain information about the job to be optimized. This information includes the job content and the amount of pre-allocated computing resources. The server is equipped with a pre-trained duration prediction model and a pre-trained scheduling optimization model for optimizing job scheduling based on the information about the job to be optimized. For ease of explanation, only the parts relevant to the embodiments of the present invention are shown.

[0120] See Figure 4 The scheduling optimization device includes:

[0121] The duration prediction module 41 is used to obtain the job content and pre-allocated computing resources of each upstream job and each downstream job in the job to be optimized, and input the job content and pre-allocated computing resources into the trained duration prediction model to obtain the predicted job duration.

[0122] The similarity calculation module 42 is used to calculate the similarity between the content of each upstream job in the job to be optimized and the content of the downstream job for any downstream job in the job to be optimized, and to obtain the target upstream job corresponding to the downstream job based on the comparison between the similarity calculation result and the preset threshold.

[0123] The timing determination module 43 is used to determine the start time of the corresponding downstream operation based on the start time of the upstream operation of each target and the predicted operation duration.

[0124] The resource scheduling module 44 is used to input the start time, predicted job duration and pre-allocated computing resources of each downstream job into the pre-trained scheduling optimization model to obtain the optimized start time and optimized computing resources of the corresponding downstream job, which is used to instruct the corresponding downstream job to be scheduled to the optimized computing resources at the optimized start time.

[0125] Optionally, the trained duration prediction model includes a trained semantic encoder and a trained fully connected layer;

[0126] The aforementioned duration prediction module 41 includes:

[0127] The semantic extraction unit is used to input the task content into the trained semantic encoder for feature extraction, and obtain a semantic feature tensor.

[0128] The feature concatenation unit is used to concatenate the embedding vector obtained by mapping computational resources with the semantic feature tensor to obtain a concatenated feature vector.

[0129] The feature mapping unit is used to input the concatenated feature vectors into the trained fully connected layer to obtain the predicted job duration.

[0130] Optionally, the similarity calculation module 42 mentioned above includes:

[0131] The keyword search unit is used to find keywords in the task content using a preset dictionary model and obtain a keyword vector.

[0132] The similarity determination unit is used to determine the cosine similarity between the keyword vectors of the corresponding upstream task and the keyword vectors of the corresponding downstream task as the similarity calculation result.

[0133] Optionally, the scheduling optimization model includes a feature encoder and a feature decoder. The sample time point, sample job duration and sample resource quantity corresponding to each historical downstream job are used as the pre-training samples of the scheduling optimization model, the actual time point and actual resource quantity corresponding to each historical downstream job are used as the pre-training labels of the scheduling optimization model, and Euclidean distance is used as the pre-training loss function.

[0134] The aforementioned scheduling optimization device also includes:

[0135] The sample feature extraction module is used to input pre-trained samples into the feature encoder for feature extraction to obtain sample feature tensors;

[0136] The sample prediction module is used to input the sample feature tensor into the feature decoder to obtain the predicted time point and predicted resource quantity for each historical downstream job.

[0137] The pre-training module is used to calculate the pre-training loss function based on the predicted time point, predicted resource amount, and pre-training labels. Based on the calculation results, the parameters of the feature encoder and feature decoder are updated through the backpropagation algorithm until the pre-training loss function converges, thus obtaining the pre-trained scheduling optimization model.

[0138] Optionally, the above-mentioned scheduling optimization device further includes:

[0139] The sample duration prediction module is used to calculate, for any historical downstream operation, the ratio of the product of the sample operation duration and the sample resource quantity of the corresponding historical downstream operation to the predicted resource quantity of the corresponding historical downstream operation, and determine the ratio as the sample prediction duration of the corresponding historical downstream operation.

[0140] The time period prediction module is used to determine the prediction time period of the corresponding historical downstream operation based on the prediction time point of the corresponding historical downstream operation and the sample prediction duration of the corresponding historical downstream operation.

[0141] The total resource calculation module is used to, for any target time point, if the predicted time period of the corresponding historical downstream operation includes the target time point, retain the corresponding historical downstream operation and determine the sum of the predicted resource amounts of all retained historical downstream operations as the total predicted resource amount for the target time point.

[0142] The first loss update module determines the maximum value in the total predicted resources for each target time point as the peak loss term, and updates the peak loss term to the pre-trained loss function by weighted summation.

[0143] Optionally, the above-mentioned scheduling optimization device further includes:

[0144] The termination point determination module is used to determine the predicted termination point of any historical downstream operation based on the predicted time period of the corresponding historical downstream operation.

[0145] The second loss update module is used to determine the maximum value of the predicted termination point corresponding to each historical downstream operation as the time loss term, and update the time loss term to the pre-trained loss function in a weighted summation manner.

[0146] Optionally, the peak loss term corresponds to the peak weight, and the time loss term corresponds to the time weight;

[0147] The aforementioned scheduling optimization device also includes:

[0148] The peak weight adjustment module is used to increase the peak weight if the peak loss term is greater than the preset resource threshold, and update the corresponding peak weight in the pre-training loss function with the increased peak weight.

[0149] The time weight adjustment module is used to increase the time weight if the time loss term is greater than a preset time threshold, and update the corresponding time weight in the pre-training loss function with the increased time weight.

[0150] It should be noted that the information interaction and execution process between the above modules and units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0151] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 The diagram shows only one of the following: a memory and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, it implements the steps in any of the above-described scheduling optimization method embodiments.

[0152] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0153] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0154] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0155] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0156] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0157] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0158] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0159] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0160] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0161] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A scheduling optimization method based on artificial intelligence, characterized in that, The method includes: Obtain the job content and pre-allocated computing resources for each upstream and downstream job in the job to be optimized. Input the job content and the pre-allocated computing resources into the trained duration prediction model to obtain the predicted job duration. For any downstream job in the job to be optimized, the similarity between the job content of each upstream job in the job to be optimized and the job content of the downstream job is calculated, and the target upstream job corresponding to the downstream job is obtained based on the comparison between the similarity calculation result and the preset threshold. Determine the start time of the corresponding downstream operation based on the start time and predicted operation duration of each upstream operation. The start time, predicted job duration, and pre-allocated computing resources of each downstream job are input into the pre-trained scheduling optimization model to obtain the optimized start time and optimized computing resources of the corresponding downstream job. This is used to instruct the corresponding downstream job to be scheduled to the optimized computing resources at the optimized start time.

2. The scheduling optimization method according to claim 1, characterized in that, The trained duration prediction model includes a trained semantic encoder and a trained fully connected layer. The step of inputting the job content and the pre-allocated computing resources into the trained duration prediction model to obtain the predicted job duration includes: The task content is input into the trained semantic encoder for feature extraction to obtain a semantic feature tensor; The embedding vector obtained by mapping the computational resources is concatenated with the semantic feature tensor to obtain the concatenated feature vector; The concatenated feature vector is input into the trained fully connected layer to obtain the predicted job duration.

3. The scheduling optimization method according to claim 1, characterized in that, The step of calculating the similarity between the content of each upstream job and the content of each downstream job in the job to be optimized includes: The keyword vector is obtained by searching for keywords in the task content using a preset dictionary model; The cosine similarity between the keyword vectors of the corresponding upstream task and the keyword vectors of the corresponding downstream task is determined as the similarity calculation result.

4. The scheduling optimization method according to any one of claims 1 to 3, characterized in that, The scheduling optimization model includes a feature encoder and a feature decoder. The sample time point, sample job duration and sample resource quantity corresponding to each historical downstream job are used as the pre-training samples of the scheduling optimization model. The actual time point and actual resource quantity corresponding to each historical downstream job are used as the pre-training labels of the scheduling optimization model. Euclidean distance is used as the pre-training loss function. The pre-training process of the scheduling optimization model includes: The pre-trained samples are input into the feature encoder for feature extraction to obtain the sample feature tensor; The sample feature tensor is input into the feature decoder to obtain the predicted time point and predicted resource quantity for each historical downstream operation; The pre-training loss function is calculated based on the predicted time point, the predicted resource amount, and the pre-trained labels. The parameters of the feature encoder and the feature decoder are updated through backpropagation based on the calculation results until the pre-training loss function converges, thus obtaining the pre-trained scheduling optimization model.

5. The scheduling optimization method according to claim 4, characterized in that, After inputting the sample feature tensor into the feature decoder and outputting the prediction time point and prediction resource quantity, the method further includes: For any historical downstream operation, calculate the ratio of the product of the sample operation duration and the sample resource quantity of the corresponding historical downstream operation to the predicted resource quantity of the corresponding historical downstream operation, and determine the ratio as the sample predicted duration of the corresponding historical downstream operation. Based on the predicted time points of the corresponding historical downstream operations and the sample predicted duration of the corresponding historical downstream operations, the predicted time period of the corresponding historical downstream operations is determined. For any target time point, if the predicted time period of the corresponding historical downstream operation includes the corresponding target time point, then the corresponding historical downstream operation is retained, and the sum of the predicted resource amounts of all retained historical downstream operations is determined to be the total predicted resource amount of the corresponding target time point. The target time point is the predicted time point among all historical downstream operations. The maximum value in the total predicted resources corresponding to each target time point is determined as the peak loss term, and the peak loss term is updated to the pre-trained loss function by weighted summation.

6. The scheduling optimization method according to claim 5, characterized in that, After determining the predicted time period for the corresponding historical downstream operations, the method further includes: For any historical downstream operation, determine the predicted termination point of the corresponding historical downstream operation based on the predicted time period of the corresponding historical downstream operation. The maximum value of the predicted termination point corresponding to each historical downstream operation is determined as the time loss term, and the time loss term is updated to the pre-trained loss function by weighted summation.

7. The scheduling optimization method according to claim 6, characterized in that, The peak loss term corresponds to the peak weight, and the time loss term corresponds to the time weight; Before calculating the pre-training loss function based on the predicted time point, the predicted resource quantity, and the pre-training labels, the method further includes: If the peak loss term is greater than the preset resource threshold, then the peak weight is increased, and the corresponding peak weight in the pre-training loss function is updated with the increased peak weight; If the time loss term is greater than a preset time threshold, the time weight is increased, and the corresponding time weight in the pre-training loss function is updated with the increased time weight.

8. A scheduling optimization device based on artificial intelligence, characterized in that, The scheduling optimization device includes: The duration prediction module is used to obtain the job content and pre-allocated computing resources of each upstream job and each downstream job in the job to be optimized, and input the job content and the pre-allocated computing resources into the trained duration prediction model to obtain the predicted job duration. The similarity calculation module is used to calculate the similarity between the content of each upstream job in the job to be optimized and the content of the downstream job for any downstream job in the job to be optimized, and to obtain the target upstream job corresponding to the downstream job based on the comparison result of the similarity calculation result and the preset threshold. The timing determination module is used to determine the start time of the corresponding downstream operation based on the start time of the upstream operation of each target and the predicted operation duration. The resource scheduling module is used to input the start time, predicted job duration, and pre-allocated computing resources of each downstream job into the pre-trained scheduling optimization model to obtain the optimized start time and optimized computing resources of the corresponding downstream job. This module is used to instruct the corresponding downstream job to be scheduled to the optimized computing resources at the optimized start time.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the scheduling optimization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the scheduling optimization method as described in any one of claims 1 to 7.

Images