Big data distributed task scheduling system and scheduling method thereof

Through the big data distributed task scheduling system, the batch operations and systems of the banking IT system are encoded semantically, and the dependencies are calculated using the attention mechanism, which solves the problems of low timeliness of batch operations and cumbersome fault investigation in the banking IT system, and realizes the More efficient task scheduling and processing order.

CN114066210BActive Publication Date: 2024-11-08WENZHOU DGM INFORMATION NETWORK ENG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111335047.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-11
Publication Date
2024-11-08
Estimated Expiration
2041-11-11

AI Technical Summary

Technical Problem

The serial execution of batch operations in banking IT systems leads to low timeliness in business volume increase, and the dependence between various IT systems is tight, resulting in cumbersome task scheduling and fault investigation.

Method used

The big data distributed task scheduling system is adopted to encode batch jobs and system descriptions through semantic understanding models, and the attention mechanism is used to calculate dependencies to achieve optimization of task scheduling and processing order.

Benefits of technology

It improves the timeliness of batch operations and the accuracy of business scheduling, and simplifies the task scheduling and fault detection process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114066210B_ABST
    Figure CN114066210B_ABST
Patent Text Reader

Abstract

The present application specifically discloses a big data distributed task scheduling system and a scheduling method thereof, which encodes the job description of each batch job in the predetermined business application system and the system description of each business application system, so that the feature vector obtained after encoding contains the dependency relationship between the batch jobs and the dependency relationship between the different business application systems, and also adopts the attention mechanism of the encoder to clip the input of the decoder based on all encoder states and the hidden state of the decoder to achieve contextual regression decoding of the decoder. In this way, the dependency relationship between the batch jobs in the same business application system and the dependency relationship between the different business application systems can be considered when determining the business scheduling and processing order, thereby making the business scheduling effect better and the timeliness of batch jobs higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of car-use services, and more specifically, to a big data distributed task scheduling system and a scheduling method thereof. Background Art

[0002] With the rapid growth of China's economy, the development speed of the banking industry has also accelerated significantly. In addition to the five major state-owned banks, Industrial and Commercial Bank of China, Agricultural Bank of China, Bank of China, and Bank of Communications, a large number of city commercial banks have been established in recent years. Although the scale of city commercial banks is smaller than that of state-owned banks, their business scope covers more than 80% of state-owned banks. Therefore, they also need IT systems to support operations. Common IT systems in the banking industry include business support, analysis and management, and regulatory reporting.

[0003] At present, batch operations of most IT systems in banks are executed serially to gradually complete daily batch operations. However, as the business volume continues to increase, the requirements for effectiveness are also getting higher and higher. At the same time, there is a relatively close dependency between various IT systems.

[0004] Currently, batch job execution in most business application systems within banks is independent of each other, which makes related operations such as configuring tasks, configuring information, viewing task dependencies, viewing task execution status and historical operation statistics extremely inconvenient. At the same time, when a batch job fails during operation, tracing the source of the problem is also very cumbersome.

[0005] Therefore, a big data distributed task scheduling system is expected to realize job-level dependent task scheduling and management system. Summary of the invention

[0006] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a big data distributed task scheduling system and a scheduling method thereof, which encodes the job description of each batch job in the predetermined business application system and the system description of each business application system, so that the feature vector obtained after encoding contains the dependency information between the batch jobs and the dependency information between the different business application systems, and also adopts the attention mechanism of the encoder to trim the input of the decoder based on all encoder states and the hidden states of the decoder to achieve contextual regression decoding of the decoder. In this way, the dependency between the batch jobs in the same business application system and the dependency between the different business application systems can be considered when determining the business scheduling and processing order, thereby making the business scheduling effect better and the timeliness of batch jobs higher.

[0007] According to one aspect of the present application, a big data distributed task scheduling system is provided, which includes:

[0008] A first semantic encoding unit, used for performing semantic encoding on the job description of each batch job in the predetermined business application system based on the semantic understanding model to obtain a sequence of job feature vectors;

[0009] A second semantic encoding unit, used for performing semantic encoding on the system description of each business application system based on the semantic understanding model to obtain a sequence of system feature vectors;

[0010] A reference feature vector selection unit, used to select a system feature vector corresponding to the predetermined business application system from the sequence of system feature vectors as a reference feature vector;

[0011] an attention unit, configured to calculate a similarity value between each job feature vector in the sequence of the job feature vectors and the reference feature vector to obtain a similarity vector composed of similarity values ​​of all the job feature vectors;

[0012] A primary decoding unit, used for passing each operation feature vector in the sequence of operation feature vectors through a decoder to obtain a category probability value of each operation feature vector in the sequence of operation feature vectors, wherein the category probability value represents a probability value of each operation feature vector belonging to an importance label with a maximum probability;

[0013] A correction coefficient determination unit, used for calculating a cross entropy value between a class probability value of each of the operation feature vectors in the operation feature vector sequence and the similarity vector as a correction coefficient of each of the operation feature vectors, so as to obtain a correction coefficient vector composed of the correction coefficients of each of the operation feature vectors;

[0014] A correction unit, used for multiplying a correction coefficient vector composed of correction coefficients of each of the operation characteristic vectors with each of the operation characteristic vectors in the operation characteristic vector sequence to obtain a sequence of corrected operation characteristic vectors composed of corrected operation characteristic vectors;

[0015] A secondary encoding unit, used for inputting each of the modified operation feature vectors in the sequence of the modified operation feature vectors into the decoder to obtain a probability value of each of the modified operation feature vectors belonging to each importance label;

[0016] an importance determination unit, configured to select, from among the probability values ​​of each of the modified job feature vectors belonging to each importance label, an importance label corresponding to a maximum probability value as the importance of the batch job corresponding to the modified job feature vector sequence; and

[0017] The scheduling result generating unit is used to determine the scheduling and processing order of each batch job based on the importance of the batch job.

[0018] In the above-mentioned big data distributed task scheduling system, the first semantic encoding unit includes: a first word embedding subunit, which is used to segment the job description of each of the jobs and pass it through the word embedding model of the semantic understanding model to obtain a sequence of input vectors corresponding to the job description of each of the jobs; and a first context encoding subunit, which is used to pass the sequence of input vectors corresponding to the job description of each of the jobs through the converter-based Bert model of the semantic understanding model to obtain a sequence of job feature vectors corresponding to the job description of each of the jobs.

[0019] In the above-mentioned big data distributed task scheduling system, the second semantic encoding unit includes: a second word embedding subunit, which is used to segment the system description of each of the business application systems and pass it through the word embedding model of the semantic understanding model to obtain a sequence of input vectors corresponding to the system description of each of the business application systems; and a second context encoding subunit, which is used to pass the sequence of input vectors corresponding to the system description of each of the business application systems through the converter-based Bert model of the semantic understanding model to obtain a sequence of job feature vectors corresponding to the system description of each of the business application systems.

[0020] In the above-mentioned big data distributed task scheduling system, the attention unit is further used to: calculate the L2 distance between each job feature vector in the sequence of the job feature vector and the reference feature vector as the similarity value.

[0021] In the above-mentioned big data distributed task scheduling system, the attention unit is further used to: calculate the cosine distance between each job feature vector in the sequence of the job feature vector and the reference feature vector as the similarity value.

[0022] In the above-mentioned big data distributed task scheduling system, the initial decoding unit is further used to: input each job feature vector in the sequence of job feature vectors into the Softmax function of the decoder to obtain the category probability value of each job feature vector in the sequence of job feature vectors.

[0023] According to another aspect of the present application, a scheduling method for a big data distributed task scheduling system includes:

[0024] Performing semantic encoding based on a semantic understanding model on the job description of each batch job in the predetermined business application system to obtain a sequence of job feature vectors;

[0025] Performing semantic coding on the system description of each business application system based on the semantic understanding model to obtain a sequence of system feature vectors;

[0026] Selecting a system feature vector corresponding to the predetermined business application system from the sequence of system feature vectors as a reference feature vector;

[0027] Calculating a similarity value between each operation feature vector in the sequence of the operation feature vectors and the reference feature vector to obtain a similarity vector composed of similarity values ​​of all the operation feature vectors;

[0028] Passing each of the job feature vectors in the sequence of the job feature vectors through a decoder to obtain a category probability value of each of the job feature vectors in the sequence of the job feature vectors, wherein the category probability value represents a probability value of each of the job feature vectors belonging to an importance label with a maximum probability;

[0029] Calculating a cross entropy value between a class probability value of each of the operation feature vectors in the operation feature vector sequence and the similarity vector as a correction coefficient of each of the operation feature vectors to obtain a correction coefficient vector composed of the correction coefficients of each of the operation feature vectors;

[0030] Multiplying a correction coefficient vector composed of correction coefficients of each of the operation characteristic vectors by each of the operation characteristic vectors in the operation characteristic vector sequence to obtain a sequence of corrected operation characteristic vectors composed of corrected operation characteristic vectors;

[0031] Inputting each of the modified operation feature vectors in the sequence of the modified operation feature vectors into the decoder to obtain a probability value of each of the modified operation feature vectors belonging to each importance label;

[0032] Selecting the importance label corresponding to the maximum probability value from the probability values ​​of each of the correction job feature vectors belonging to each importance label as the importance of the batch job corresponding to the correction job feature vector sequence; and

[0033] Based on the importance of the batch jobs, the scheduling and processing order of the batch jobs are determined.

[0034] In the scheduling method of the above-mentioned big data distributed task scheduling system, the job description of each batch job in the predetermined business application system is semantically encoded based on the semantic understanding model to obtain a sequence of job feature vectors, including: after the job description of each of the jobs is segmented, the word embedding model of the semantic understanding model is used to obtain a sequence of input vectors corresponding to the job description of each of the jobs; and the sequence of input vectors corresponding to the job description of each of the jobs is passed through the converter-based Bert model of the semantic understanding model to obtain a sequence of job feature vectors corresponding to the job description of each of the jobs.

[0035] In the scheduling method of the big data distributed task scheduling system, the system description of each business application system is semantically encoded based on the semantic understanding model to obtain a sequence of system feature vectors, including: after the system description of each business application system is segmented, the word embedding model of the semantic understanding model is used to obtain a sequence of input vectors corresponding to the system description of each business application system; and the sequence of input vectors corresponding to the system description of each business application system is passed through the converter-based Bert model of the semantic understanding model to obtain a sequence of job feature vectors corresponding to the system description of each business application system.

[0036] In the scheduling method of the above-mentioned big data distributed task scheduling system, the similarity value between each job feature vector in the sequence of the job feature vector and the reference feature vector is calculated to obtain a similarity vector composed of the similarity values ​​of all the job feature vectors, including: calculating the L2 distance between each job feature vector in the sequence of the job feature vector and the reference feature vector as the similarity value.

[0037] In the scheduling method of the above-mentioned big data distributed task scheduling system, the similarity value between each job feature vector in the sequence of the job feature vector and the reference feature vector is calculated to obtain a similarity vector composed of the similarity values ​​of all the job feature vectors, including: calculating the cosine distance between each job feature vector in the sequence of the job feature vector and the reference feature vector as the similarity value.

[0038] In the scheduling method of the above-mentioned big data distributed task scheduling system, each job feature vector in the sequence of the job feature vectors is passed through a decoder to obtain a category probability value of each of the job feature vectors in the sequence of the job feature vectors, and the category probability value represents the probability value of each of the job feature vectors belonging to the importance label with the maximum probability, including: each job feature vector in the sequence of the job feature vectors is input into the Softmax function of the decoder to obtain the category probability value of each of the job feature vectors in the sequence of the job feature vectors.

[0039] Compared with the prior art, the big data distributed task scheduling system and scheduling method provided by the present application encode the job description of each batch job in the predetermined business application system and the system description of each business application system, so that the feature vector obtained after encoding contains the information of the dependency relationship between the batch jobs and the dependency relationship between the different business application systems, and also adopts the attention mechanism of the encoder to trim the input of the decoder based on all encoder states and the hidden state of the decoder to realize the context regression decoding of the decoder. In this way, the dependency relationship between the batch jobs in the same business application system and the dependency relationship between the different business application systems can be considered when determining the business scheduling and processing order, thereby making the business scheduling effect better and the timeliness of batch jobs higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0041] Figure 1 This is an application scenario diagram of a big data distributed task scheduling system according to an embodiment of the present application.

[0042] Figure 2 4 is a block diagram of a big data distributed task scheduling system according to an embodiment of the present application.

[0043] Figure 3 The present invention is a flowchart of a scheduling method for a big data distributed task scheduling system according to an embodiment of the present application.

[0044] Figure 4 Schematic diagram of the architecture of a scheduling method for a big data distributed task scheduling system according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0046] Scenario Overview

[0047] As mentioned above, most of the batch jobs in the IT systems of banks are currently executed in serial mode, and the daily batch jobs are gradually completed. However, with the continuous increase in business volume, the requirements for effectiveness are also getting higher and higher. At the same time, there is a relatively close dependency between various IT systems.

[0048] At present, the batch job execution of most business application systems in banks is independent of each other, which makes it very inconvenient to configure tasks, configure information, view task dependencies, view task execution status and historical operation statistics. At the same time, when a batch job fails during operation, it is also very cumbersome to trace the source of the problem. Therefore, a big data distributed task scheduling system is expected to realize the task scheduling and management system with job-level dependencies.

[0049] Accordingly, in the technical solution of the present application, the job description of each batch job in the predetermined business application system is semantically encoded based on a semantic understanding model, that is, each job description is converted into an input vector through word embedding, and then the sequence of input vectors corresponding to each job description is passed through a semantic understanding model, such as a transformer-based Bert model, to obtain a sequence of job feature vectors.

[0050] In addition, the system description of each business application system is semantically encoded based on a semantic understanding model. That is, each system description is converted into an input vector through word embedding, and then the sequence of input vectors corresponding to each system description is passed through a semantic understanding model, such as a transformer-based Bert model, to obtain a sequence of system feature vectors.

[0051] A predetermined system feature vector corresponding to the predetermined business application system is selected as a reference vector, and the encoder attention mechanism is applied to introduce the context mechanism into the decoder, that is, the similarity value between each job feature vector in the sequence of job feature vectors and the predetermined system feature vector is calculated, and a similarity vector is formed for all job feature vectors, and then the category probability value obtained after each job feature vector in the sequence of job feature vectors passes through the decoder is calculated, that is, the probability value of the importance label with the maximum probability, and the cross entropy value between the similarity vector and the category probability value is calculated as the correction coefficient of each job feature vector.

[0052] Finally, the correction coefficient is multiplied by the job feature vector and then input into the decoder again to obtain the probability value belonging to each importance label, and the importance label corresponding to the maximum probability value is selected to determine the importance of the batch job.

[0053] Based on this, the present application proposes a big data distributed task scheduling system, which includes: a first semantic encoding unit, which is used to perform semantic encoding on the job description of each batch job in the predetermined business application system based on a semantic understanding model to obtain a sequence of job feature vectors; a second semantic encoding unit, which is used to perform semantic encoding on the system description of each business application system based on the semantic understanding model to obtain a sequence of system feature vectors; a reference feature vector selection unit, which is used to select the system feature vector corresponding to the predetermined business application system from the sequence of system feature vectors as a reference feature vector; an attention unit, which is used to calculate the similarity value between each job feature vector in the sequence of job feature vectors and the reference feature vector to obtain a similarity vector composed of the similarity values ​​of all the job feature vectors; a preliminary decoding unit, which is used to pass each job feature vector in the sequence of job feature vectors through a decoder to obtain a category probability value of each job feature vector in the sequence of job feature vectors, wherein the category probability value represents the probability value of each job feature vector belonging to the importance label with the maximum probability; a correction coefficient A determination unit, used to calculate the cross entropy value between the category probability value of each of the job feature vectors in the job feature vector sequence and the similarity vector as the correction coefficient of each of the job feature vectors, so as to obtain a correction coefficient vector composed of the correction coefficients of each of the job feature vectors; a correction unit, used to multiply the correction coefficient vector composed of the correction coefficients of each of the job feature vectors with each of the job feature vectors in the job feature vector sequence to obtain a sequence of corrected job feature vectors composed of corrected job feature vectors; a secondary encoding unit, used to input each of the corrected job feature vectors in the sequence of corrected job feature vectors into the decoder to obtain the probability value of each of the corrected job feature vectors belonging to each importance label; an importance determination unit, used to select the importance label corresponding to the maximum probability value from the probability values ​​of each of the corrected job feature vectors belonging to each importance label as the importance of the batch job corresponding to the corrected job feature vector sequence; and a scheduling result generation unit, used to determine the scheduling and processing order of each of the batch jobs based on the importance of the batch jobs.

[0054] Figure 1 The diagram shows an application scenario diagram of a big data distributed task scheduling system according to an embodiment of the present application. Figure 1 As shown, in this application scenario, first, from the predetermined business application system in the bank (for example, Figure 1 B1) shown in FIG. 1 obtains each batch job (for example, Figure 1 W1 to Wn) as shown in the figure, and obtaining the job description of each business application system (for example, Figure 1Then, the obtained job descriptions of each batch job in the predetermined business application system and the system descriptions of each business application system are input into a server deployed with a big data distributed task scheduling algorithm (for example, Figure 1 The cloud server S shown in the figure), wherein the server can process the job description of each batch job in the predetermined business application system and the system description of each business application system with a big data distributed task scheduling algorithm to generate a probability value of each correction job feature vector belonging to each importance label. Then, the importance label corresponding to the maximum probability value in the probability value is selected as the importance of the batch job corresponding to the correction job feature vector sequence, and then, based on the importance of the batch job, the scheduling and processing order of each batch job is determined to achieve a better effect of business scheduling and a higher timeliness of batch jobs.

[0055] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0056] Exemplary Systems

[0057] Figure 2 The diagram shows a block diagram of a big data distributed task scheduling system according to an embodiment of the present application. Figure 2As shown, according to the big data distributed task scheduling system 200 of the embodiment of the present application, it includes: a first semantic encoding unit 210, which is used to perform semantic encoding based on a semantic understanding model on the job description of each batch job in the predetermined business application system to obtain a sequence of job feature vectors; a second semantic encoding unit 220, which is used to perform semantic encoding based on the semantic understanding model on the system description of each business application system to obtain a sequence of system feature vectors; a reference feature vector selection unit 230, which is used to select the system feature vector corresponding to the predetermined business application system from the sequence of system feature vectors as a reference feature vector; an attention unit 240, which is used to calculate the similarity value between each job feature vector in the sequence of job feature vectors and the reference feature vector to obtain a similarity vector composed of the similarity values ​​of all the job feature vectors; a preliminary decoding unit 250, which is used to pass each job feature vector in the sequence of job feature vectors through a decoder to obtain a category probability value of each job feature vector in the sequence of job feature vectors, wherein the category probability value represents the probability value of each job feature vector belonging to the importance label with the maximum probability; a correction system A number determination unit 260 is used to calculate the cross entropy value between the category probability value of each of the job feature vectors in the job feature vector sequence and the similarity vector as the correction coefficient of each of the job feature vectors to obtain a correction coefficient vector composed of the correction coefficients of each of the job feature vectors; a correction unit 270 is used to multiply the correction coefficient vector composed of the correction coefficients of each of the job feature vectors with each of the job feature vectors in the job feature vector sequence to obtain a sequence of corrected job feature vectors composed of corrected job feature vectors; a secondary encoding unit 280 is used to input each of the corrected job feature vectors in the sequence of corrected job feature vectors into the decoder to obtain a probability value of each of the corrected job feature vectors belonging to each importance label; an importance determination unit 290 is used to select the importance label corresponding to the maximum probability value from the probability values ​​of each of the corrected job feature vectors belonging to each importance label as the importance of the batch job corresponding to the corrected job feature vector sequence; and a scheduling result generation unit 300 is used to determine the scheduling and processing order of each of the batch jobs based on the importance of the batch jobs.

[0058] Specifically, in the embodiment of the present application, the first semantic encoding unit 210 and the second semantic encoding unit 220 are used to semantically encode the job description of each batch job in the predetermined business application system based on the semantic understanding model to obtain a sequence of job feature vectors, and to semantically encode the system description of each business application system based on the semantic understanding model to obtain a sequence of system feature vectors. As mentioned above, in order to perform business scheduling of batch jobs, it is necessary to determine the business level of the batch jobs, so as to perform business scheduling based on the business level. However, when determining the business level, it is necessary to consider the dependencies between the batch jobs in the same business application system and the dependencies between the different business application systems.

[0059] Therefore, in the technical solution of the present application, first, the job description of each batch job is obtained from the predetermined business application system in the bank, and the system description of each business application system is obtained. Then, the job description of each batch job in the predetermined business application system is semantically encoded based on the semantic understanding model, that is, each of the job descriptions is converted into an input vector through word embedding, and then the sequence of input vectors corresponding to each job description is passed through a semantic understanding model, such as a converter-based Bert model, to obtain a sequence of job feature vectors. Then, the system description of each business application system is also semantically encoded based on the semantic understanding model, that is, each of the system descriptions is converted into an input vector through word embedding, and then the sequence of input vectors corresponding to each system description is passed through a semantic understanding model, such as a converter-based Bert model, to obtain a sequence of system feature vectors. It should be understood that since the current semantic understanding model can be encoded based on contextual relationships, by encoding the job descriptions of each batch job in the predetermined business application system and the system descriptions of each business application system, the feature vector obtained after encoding can include information on the dependencies between the batch jobs and the dependencies between the different business application systems, so that the scheduling results generated subsequently are more accurate.

[0060] More specifically, in an embodiment of the present application, the first semantic encoding unit includes: a first word embedding subunit, used to segment the job description of each of the jobs and then pass it through the word embedding model of the semantic understanding model to obtain a sequence of input vectors corresponding to the job description of each of the jobs; and a first context encoding subunit, used to pass the sequence of input vectors corresponding to the job description of each of the jobs through the converter-based Bert model of the semantic understanding model to obtain a sequence of job feature vectors corresponding to the job description of each of the jobs.

[0061] More specifically, in an embodiment of the present application, the second semantic encoding unit includes: a second word embedding subunit, used to segment the system description of each of the business application systems and then pass it through the word embedding model of the semantic understanding model to obtain a sequence of input vectors corresponding to the system description of each of the business application systems; and a second context encoding subunit, used to pass the sequence of input vectors corresponding to the system description of each of the business application systems through the converter-based Bert model of the semantic understanding model to obtain a sequence of job feature vectors corresponding to the system description of each of the business application systems.

[0062] Specifically, in the embodiment of the present application, the reference feature vector selection unit 230 and the attention unit 240 are used to select the system feature vector corresponding to the predetermined business application system from the sequence of the system feature vectors as the reference feature vector, and calculate the similarity value between each job feature vector in the sequence of the job feature vectors and the reference feature vector to obtain a similarity vector composed of the similarity values ​​of all the job feature vectors. That is, in the technical solution of the present application, first, the predetermined system feature vector corresponding to the predetermined business application system is selected from the sequence of the system feature vectors as the reference vector. Then, the encoder attention mechanism is applied to introduce the context mechanism into the decoder to bring a contextual relationship to the encoder state. Specifically, in the embodiment of the present application, the similarity value between each job feature vector in the sequence of the job feature vectors and the predetermined system feature vector is first calculated, and then the similarity values ​​of all job feature vectors are used to form a similarity vector.

[0063] More specifically, in a specific example, the L2 distance between each job feature vector in the sequence of the job feature vector and the reference feature vector can be calculated as the similarity value. It should be understood that a person skilled in the art should understand that the L2 distance function, also known as the least square error (LSE), is the sum of the squares of the difference between the target value and the estimated value, also known as the Euclidean distance, and the formula is D = |x i -y i | 2 , x i represents the characteristic value of each position in each operation feature vector in the sequence of the operation feature vectors, y i Represents the eigenvalues ​​of each position in the reference feature vector. It should be understood that, compared with the L1 distance function, the L2 distance function has a stable solution. Here, calculating the L2 distance between the eigenvalues ​​of each position of each operation feature vector in the sequence of the operation feature vector and the reference feature vector can reflect the degree of feature difference between each operation feature vector in the sequence of the operation feature vector and the reference feature vector.

[0064] In particular, in another specific example, the cosine distance between each job feature vector in the sequence of the job feature vector and the reference feature vector can also be calculated as the similarity value. It should be understood that cosine similarity is to calculate the cosine value of the angle between two vectors, that is, for two vectors A and B, the cosine similarity is defined as: The cosine distance is 1 minus the cosine similarity, that is, it can be defined by the formula:

[0065]

[0066] And since the value range of the cosine similarity is [-1, 1], the value range of the cosine distance is [0, 2]. It should be understood that the Euclidean distance reflects the absolute difference in distance, while the cosine distance reflects the relative difference in direction.

[0067] Specifically, in the embodiment of the present application, the initial decoding unit 250 is used to pass each operation feature vector in the sequence of the operation feature vectors through a decoder to obtain a category probability value of each operation feature vector in the sequence of the operation feature vectors, and the category probability value represents the probability value of each operation feature vector belonging to the importance label with the maximum probability. That is, in the technical solution of the present application, for the similarity vector composed of all the operation feature vectors, the category probability value obtained after each operation feature vector in the sequence of the operation feature vectors passes through the decoder is calculated, and here, the category probability value represents the probability value of each operation feature vector belonging to the importance label with the maximum probability.

[0068] In a specific example, the initial decoding unit is further used to: input each operation feature vector in the sequence of operation feature vectors into the Softmax function of the decoder to obtain a category probability value of each operation feature vector in the sequence of operation feature vectors.

[0069] Specifically, in the embodiment of the present application, the correction coefficient determination unit 260 and the correction unit 270 are used to calculate the cross entropy value between the category probability value of each of the job feature vectors in the job feature vector sequence and the similarity vector as the correction coefficient of each of the job feature vectors, so as to obtain a correction coefficient vector composed of the correction coefficients of each of the job feature vectors, and multiply the correction coefficient vector composed of the correction coefficients of each of the job feature vectors with each of the job feature vectors in the job feature vector sequence to obtain a sequence of corrected job feature vectors composed of corrected job feature vectors. It should be understood that in the technical solution of the present application, in order to correct each of the job feature vectors in the job feature vector sequence, after obtaining the probability value of each of the job feature vectors belonging to the importance label with the maximum probability, first, the cross entropy value between the category probability value of each of the job feature vectors in the job feature vector sequence and the similarity vector is calculated as the correction coefficient of each of the job feature vectors, and the obtained correction coefficients are formed into a correction coefficient vector for subsequent correction calculation. Then, a correction coefficient vector composed of correction coefficients of each of the job feature vectors is multiplied with each of the job feature vectors in the job feature vector sequence to correct each of the job feature vectors in the job feature vector sequence, thereby obtaining a sequence of corrected job feature vectors composed of corrected job feature vectors. In this way, the decoder regression decoding of each batch job can further refer to the dependency relationship between different business application systems and the contextual relationship brought to the encoder state on the basis of the dependency relationship between the batch jobs, so as to make the scheduling of each batch job more accurate.

[0070] Specifically, in the embodiment of the present application, the secondary encoding unit 280 is used to input each of the corrected job feature vectors in the sequence of the corrected job feature vectors into the decoder to obtain the probability value of each of the corrected job feature vectors belonging to each importance label. That is, in the technical solution of the present application, after correcting each of the job feature vectors in the sequence of the job feature vectors, each of the corrected job feature vectors in the sequence of the corrected job feature vectors is input into the decoder to obtain the probability value of each of the corrected job feature vectors belonging to each importance label. It should be understood that through the encoder attention mechanism, the decoder regression decoding of each batch job can be further referenced to the dependency relationship between different business application systems and the contextual relationship brought to the encoder state on the basis of the dependency relationship between the batch jobs, that is, the input of the decoder is trimmed based on all the encoder states and the hidden state of the decoder to achieve the contextual regression decoding of the decoder.

[0071] Specifically, in the embodiment of the present application, the importance determination unit 290 and the scheduling result generation unit 300 are used to select the importance label corresponding to the maximum probability value from the probability values ​​of each of the modified job feature vectors belonging to each importance label as the importance of the batch job corresponding to the modified job feature vector sequence, and determine the scheduling and processing order of each batch job based on the importance of the batch job. It should be understood that in the technical solution of the present application, the dependencies between the batch jobs in the same business application system and the dependencies between the different business application systems can be considered when determining the business level, so as to determine the business importance level of the batch job, and then determine the scheduling and processing order of each batch job based on the importance of the batch job. In this way, the accuracy of business scheduling can be higher and the timeliness of batch jobs can be higher.

[0072] In summary, the big data distributed task scheduling system 200 based on the embodiment of the present application is explained, which encodes the job description of each batch job in the predetermined business application system and the system description of each business application system, so that the feature vector obtained after encoding contains the information of the dependency relationship between the batch jobs and the dependency relationship between the different business application systems, and also adopts the attention mechanism of the encoder to trim the input of the decoder based on all encoder states and the hidden state of the decoder to achieve contextual regression decoding of the decoder. In this way, the dependency relationship between the batch jobs in the same business application system and the dependency relationship between the different business application systems can be considered when determining the business scheduling and processing order, thereby making the business scheduling effect better and the timeliness of batch jobs higher.

[0073] As described above, the big data distributed task scheduling system 200 according to the embodiment of the present application can be implemented in various terminal devices, such as a server of a big data distributed task scheduling algorithm. In one example, the big data distributed task scheduling system 200 according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the big data distributed task scheduling system 200 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the big data distributed task scheduling system 200 can also be one of the many hardware modules of the terminal device.

[0074] Alternatively, in another example, the big data distributed task scheduling system 200 and the terminal device may also be separate devices, and the big data distributed task scheduling system 200 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0075] Exemplary Methods

[0076] Figure 3 The figure shows a flow chart of the scheduling method of the big data distributed task scheduling system. Figure 3 As shown, according to the scheduling method of the big data distributed task scheduling system of the embodiment of the present application, the steps include: S110, semantically encoding the job description of each batch job in the predetermined business application system based on the semantic understanding model to obtain a sequence of job feature vectors; S120, semantically encoding the system description of each business application system based on the semantic understanding model to obtain a sequence of system feature vectors; S130, selecting the system feature vector corresponding to the predetermined business application system from the sequence of system feature vectors as a reference feature vector; S140, calculating the similarity value between each job feature vector in the sequence of job feature vectors and the reference feature vector to obtain a similarity vector composed of the similarity values ​​of all the job feature vectors; S150, passing each job feature vector in the sequence of job feature vectors through a decoder to obtain a category probability value of each job feature vector in the sequence of job feature vectors, wherein the category probability value represents the probability value of each job feature vector belonging to the importance label with the maximum probability; S16 S170, multiplying the correction coefficient vector composed of the correction coefficients of each job feature vector with each job feature vector in the job feature vector sequence to obtain a sequence of corrected job feature vectors composed of corrected job feature vectors; S180, inputting each corrected job feature vector in the sequence of corrected job feature vectors into the decoder to obtain the probability value of each corrected job feature vector belonging to each importance label; S190, selecting the importance label corresponding to the maximum probability value from the probability values ​​of each corrected job feature vector belonging to each importance label as the importance of the batch job corresponding to the corrected job feature vector sequence; and, S200, determining the scheduling and processing order of each batch job based on the importance of the batch job.

[0077] Figure 4 The diagram shows a schematic diagram of the architecture of the scheduling method of the big data distributed task scheduling system according to the embodiment of the present application. Figure 4 As shown, in the network architecture of the scheduling method of the big data distributed task scheduling system, first, the job description of each batch job in the predetermined business application system is obtained (for example, Figure 4IN1) is based on a semantic understanding model (e.g., Figure 4 SUM) to obtain a sequence of job feature vectors (e.g., Figure 4 VF1 shown in FIG); then, the system description of each business application system obtained (for example, Figure 4 IN2) is performed based on the semantic understanding model (for example, Figure 4 SUM) to obtain a sequence of system feature vectors (e.g., Figure 4 Then, the system feature vector corresponding to the predetermined business application system is selected from the sequence of system feature vectors as a reference feature vector (for example, Figure 4 VF3 shown in FIG3 ); then, calculating the similarity value between each operation feature vector in the sequence of the operation feature vector and the reference feature vector (for example, Figure 4 SV shown in the figure) to obtain a similarity vector composed of similarity values ​​of all the job feature vectors (for example, Figure 4 Then, each job feature vector in the sequence of job feature vectors is passed through a decoder (for example, Figure 4 D) to obtain the class probability value of each of the job feature vectors in the job feature vector sequence (for example, Figure 4 Then, the cross entropy value between the class probability value of each of the operation feature vectors in the operation feature vector sequence and the similarity vector is calculated as the correction coefficient of each of the operation feature vectors (for example, Figure 4 CF) to obtain a correction coefficient vector composed of correction coefficients of each of the operation feature vectors (for example, Figure 4 Then, a correction coefficient vector composed of correction coefficients of each of the operation feature vectors is multiplied by each of the operation feature vectors in the operation feature vector sequence to obtain a sequence of corrected operation feature vectors composed of corrected operation feature vectors (for example, Figure 4 Then, each of the modified operation feature vectors in the sequence of the modified operation feature vectors is input into the decoder (for example, Figure 4 D) to obtain the probability value of each correction operation feature vector belonging to each importance label (for example, Figure 4PV shown in ); then, selecting the importance label corresponding to the maximum probability value from the probability values ​​of each of the modified job feature vectors belonging to each importance label as the importance of the batch job corresponding to the modified job feature vector sequence; and, finally, determining the scheduling and processing order of each batch job based on the importance of the batch job.

[0078] More specifically, in steps S110 and S120, semantic encoding based on a semantic understanding model is performed on the job description of each batch job in the predetermined business application system to obtain a sequence of job feature vectors, and semantic encoding based on the semantic understanding model is performed on the system description of each business application system to obtain a sequence of system feature vectors. That is, first, the job description of each batch job is obtained from the predetermined business application system in the bank, and the system description of each business application system is obtained. Next, semantic encoding based on a semantic understanding model is performed on the job description of each batch job in the predetermined business application system, that is, each of the job descriptions is converted into an input vector through word embedding, and then the sequence of input vectors corresponding to each job description is passed through a semantic understanding model, such as a Bert model based on a converter, to obtain a sequence of job feature vectors. Then, semantic encoding based on a semantic understanding model is also performed on the system description of each business application system, that is, each of the system descriptions is converted into an input vector through word embedding, and then the sequence of input vectors corresponding to each system description is passed through a semantic understanding model, such as a Bert model based on a converter, to obtain a sequence of system feature vectors. It should be understood that since the current semantic understanding model can be encoded based on contextual relationships, by encoding the job descriptions of each batch job in the predetermined business application system and the system descriptions of each business application system, the feature vector obtained after encoding can include information on the dependencies between the batch jobs and the dependencies between the different business application systems, so that the scheduling results generated subsequently are more accurate.

[0079] More specifically, in steps S130 and S140, the system feature vector corresponding to the predetermined business application system is selected from the sequence of the system feature vectors as a reference feature vector, and the similarity value between each job feature vector in the sequence of the job feature vectors and the reference feature vector is calculated to obtain a similarity vector composed of the similarity values ​​of all the job feature vectors. That is, in the technical solution of the present application, first, a predetermined system feature vector corresponding to the predetermined business application system is selected from the sequence of the system feature vectors as a reference vector. Then, the encoder attention mechanism is applied to introduce the context mechanism into the decoder to bring a contextual relationship to the encoder state. Specifically, in an embodiment of the present application, the similarity value between each job feature vector in the sequence of the job feature vectors and the predetermined system feature vector is first calculated, and then the similarity values ​​of all job feature vectors are used to form a similarity vector.

[0080] More specifically, in step S150, each job feature vector in the sequence of job feature vectors is passed through a decoder to obtain a category probability value for each of the job feature vectors in the sequence of job feature vectors, and the category probability value represents the probability value of each of the job feature vectors belonging to the importance label with the maximum probability. That is, in the technical solution of the present application, for the similarity vector composed of all the job feature vectors, the category probability value obtained after each of the job feature vectors in the sequence of job feature vectors passes through the decoder is calculated, and here, the category probability value represents the probability value of each of the job feature vectors belonging to the importance label with the maximum probability. In a specific example, each of the job feature vectors in the sequence of job feature vectors can be input into the Softmax function of the decoder to obtain the category probability value of each of the job feature vectors in the sequence of job feature vectors.

[0081] More specifically, in step S160 and step S170, the cross entropy value between the category probability value of each of the operation feature vectors in the operation feature vector sequence and the similarity vector is calculated as the correction coefficient of each of the operation feature vectors to obtain a correction coefficient vector composed of the correction coefficients of each of the operation feature vectors, and the correction coefficient vector composed of the correction coefficients of each of the operation feature vectors is multiplied with each of the operation feature vectors in the operation feature vector sequence to obtain a sequence of corrected operation feature vectors composed of corrected operation feature vectors. That is, after obtaining the probability value of each of the operation feature vectors belonging to the importance label with the maximum probability, first, the cross entropy value between the category probability value of each of the operation feature vectors in the operation feature vector sequence and the similarity vector is calculated as the correction coefficient of each of the operation feature vectors, and the obtained correction coefficients are combined into a correction coefficient vector for subsequent correction calculations. Then, a correction coefficient vector composed of correction coefficients of each of the job feature vectors is multiplied with each of the job feature vectors in the job feature vector sequence to correct each of the job feature vectors in the job feature vector sequence, thereby obtaining a sequence of corrected job feature vectors composed of corrected job feature vectors. In this way, the decoder regression decoding of each batch job can further refer to the dependency relationship between different business application systems and the contextual relationship brought to the encoder state on the basis of the dependency relationship between the batch jobs, so as to make the scheduling of each batch job more accurate.

[0082] More specifically, in step S180, each of the corrected job feature vectors in the sequence of corrected job feature vectors is input into the decoder to obtain the probability value of each of the corrected job feature vectors belonging to each importance label. That is, after correcting each of the job feature vectors in the sequence of job feature vectors, each of the corrected job feature vectors in the sequence of corrected job feature vectors is input into the decoder to obtain the probability value of each of the corrected job feature vectors belonging to each importance label. It should be understood that through the encoder attention mechanism, the decoder regression decoding of each batch job can further refer to the dependency relationship between different business application systems and the contextual relationship brought to the encoder state on the basis of the dependency relationship between the batch jobs, that is, the input of the decoder is trimmed based on all the encoder states and the hidden state of the decoder to achieve the contextual regression decoding of the decoder.

[0083] More specifically, in step S190 and step S200, the importance label corresponding to the maximum probability value is selected from the probability values ​​of each of the modified job feature vectors belonging to each importance label as the importance of the batch job corresponding to the modified job feature vector sequence, and based on the importance of the batch job, the scheduling and processing order of each batch job is determined. It should be understood that in the technical solution of the present application, the dependencies between the batch jobs in the same business application system and the dependencies between the different business application systems can be considered when determining the business level, so as to determine the business importance level of the batch job, and then determine the scheduling and processing order of each batch job based on the importance of the batch job. In this way, the accuracy of business scheduling can be higher and the timeliness of batch jobs can be higher.

[0084] In summary, the scheduling method of the big data distributed task scheduling system based on the embodiment of the present application is explained, which encodes the job description of each batch job in the predetermined business application system and the system description of each business application system, so that the feature vector obtained after encoding contains the dependency information between the batch jobs and the dependency information between the different business application systems, and also adopts the attention mechanism of the encoder to trim the input of the decoder based on all encoder states and the hidden states of the decoder to achieve contextual regression decoding of the decoder. In this way, the dependency between the batch jobs in the same business application system and the dependency between the different business application systems can be considered when determining the business scheduling and processing order, thereby making the business scheduling effect better and the timeliness of batch jobs higher.

[0085] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0086] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0087] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0088] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0089] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A big data distributed task scheduling system, characterized in that: include: A first semantic encoding unit, used for performing semantic encoding on the job description of each batch job in the predetermined business application system based on the semantic understanding model to obtain a sequence of job feature vectors; A second semantic encoding unit, used for performing semantic encoding on the system description of each business application system based on the semantic understanding model to obtain a sequence of system feature vectors; A reference feature vector selection unit, used to select a system feature vector corresponding to the predetermined business application system from the sequence of system feature vectors as a reference feature vector; an attention unit, configured to calculate a similarity value between each job feature vector in the sequence of job feature vectors and the reference feature vector to obtain a similarity vector composed of similarity values ​​of all job feature vectors; A primary decoding unit, used for passing each operation feature vector in the sequence of the operation feature vectors through a decoder to obtain a category probability value of each operation feature vector in the sequence of the operation feature vectors, wherein the category probability value represents a probability value of each operation feature vector belonging to an importance label with a maximum probability; A correction coefficient determination unit, used for calculating a cross entropy value between a class probability value of each of the operation feature vectors in the sequence of the operation feature vectors and the similarity vector as a correction coefficient of each of the operation feature vectors, so as to obtain a correction coefficient vector composed of the correction coefficients of each of the operation feature vectors; A correction unit, used for multiplying a correction coefficient vector composed of correction coefficients of each of the operation characteristic vectors with each of the operation characteristic vectors in the sequence of the operation characteristic vectors to obtain a sequence of corrected operation characteristic vectors composed of corrected operation characteristic vectors; A secondary encoding unit, used for inputting each of the modified operation feature vectors in the sequence of the modified operation feature vectors into the decoder to obtain a probability value of each of the modified operation feature vectors belonging to each importance label; An importance determination unit, configured to select the importance label corresponding to the maximum probability value from the probability values ​​of each of the correction job feature vectors belonging to each importance label as the importance of the batch job corresponding to the sequence of the correction job feature vectors; as well as A scheduling result generating unit, used for determining the scheduling and processing order of each batch job based on the importance of the batch job; Wherein, the first semantic encoding unit includes: A first word embedding subunit, configured to segment the job description of each of the jobs and then pass it through the word embedding model of the semantic understanding model to obtain a sequence of input vectors corresponding to the job description of each of the jobs; and A first context encoding subunit, configured to pass a sequence of input vectors corresponding to the job description of each of the jobs through a converter-based Bert model of the semantic understanding model to obtain a sequence of job feature vectors corresponding to the job description of each of the jobs; Wherein, the second semantic encoding unit includes: A second word embedding subunit is used to segment the system description of each of the business application systems and pass it through the word embedding model of the semantic understanding model to obtain a sequence of input vectors corresponding to the system description of each of the business application systems; and The second context encoding subunit is used to obtain the sequence of job feature vectors corresponding to the system description of each business application system through the converter-based Bert model of the semantic understanding model.

2. The big data distributed task scheduling system according to claim 1, wherein: The attention unit is further used to calculate the Euclidean distance between each job feature vector in the sequence of job feature vectors and the reference feature vector as the similarity value.

3. The big data distributed task scheduling system according to claim 2, wherein: The attention unit is further used to calculate the cosine distance between each job feature vector in the sequence of job feature vectors and the reference feature vector as the similarity value.

4. The big data distributed task scheduling system according to claim 2 or 3, wherein: The initial decoding unit is further used to: input each operation feature vector in the sequence of operation feature vectors into the Softmax function of the decoder to obtain a category probability value of each operation feature vector in the sequence of operation feature vectors.

5. A scheduling method for a big data distributed task scheduling system, characterized in that: include: Performing semantic encoding based on a semantic understanding model on the job description of each batch job in the predetermined business application system to obtain a sequence of job feature vectors; Performing semantic coding on the system description of each business application system based on the semantic understanding model to obtain a sequence of system feature vectors; Selecting a system feature vector corresponding to the predetermined business application system from the sequence of system feature vectors as a reference feature vector; Calculating a similarity value between each operation feature vector in the sequence of the operation feature vectors and the reference feature vector to obtain a similarity vector composed of similarity values ​​of all operation feature vectors; Passing each of the operation feature vectors in the sequence of the operation feature vectors through a decoder to obtain a category probability value of each of the operation feature vectors in the sequence of the operation feature vectors, wherein the category probability value represents a probability value of each of the operation feature vectors belonging to an importance label with a maximum probability; Calculating a cross entropy value between a class probability value of each of the operation feature vectors in the sequence of the operation feature vectors and the similarity vector as a correction coefficient of each of the operation feature vectors to obtain a correction coefficient vector composed of the correction coefficients of each of the operation feature vectors; Multiplying each of the operation feature vectors in the sequence of the operation feature vectors with a correction coefficient vector composed of correction coefficients of the respective operation feature vectors to obtain a sequence of corrected operation feature vectors composed of corrected operation feature vectors; Inputting each of the modified operation feature vectors in the sequence of the modified operation feature vectors into the decoder to obtain a probability value of each of the modified operation feature vectors belonging to each importance label; Selecting the importance label corresponding to the maximum probability value from the probability values ​​of each of the modified job feature vectors belonging to each importance label as the importance of the batch job corresponding to the sequence of the modified job feature vectors; as well as Determining the scheduling and processing order of each batch job based on the importance of the batch jobs; The operation description of each batch operation in the predetermined business application system is semantically encoded based on the semantic understanding model to obtain a sequence of operation feature vectors, including: After the job description of each of the jobs is segmented, the word embedding model of the semantic understanding model is used to obtain a sequence of input vectors corresponding to the job description of each of the jobs; and Passing the sequence of input vectors corresponding to the job description of each job through the converter-based Bert model of the semantic understanding model to obtain the sequence of job feature vectors corresponding to the job description of each job; The system description of each business application system is semantically encoded based on the semantic understanding model to obtain a sequence of system feature vectors, including: After the system description of each of the business application systems is segmented, the word embedding model of the semantic understanding model is used to obtain a sequence of input vectors corresponding to the system description of each of the business application systems; and The sequence of input vectors corresponding to the system description of each business application system is passed through the converter-based Bert model of the semantic understanding model to obtain the sequence of job feature vectors corresponding to the system description of each business application system.

6. The scheduling method of the big data distributed task scheduling system according to claim 5, wherein: Calculating the similarity value between each job feature vector in the sequence of the job feature vectors and the reference feature vector to obtain a similarity vector composed of the similarity values ​​of all the job feature vectors, including: The Euclidean distance between each operation feature vector in the sequence of the operation feature vectors and the reference feature vector is calculated as the similarity value.