Intelligent submission method and system for big data jobs in container clusters

Abstract

The present invention provides a method and system for intelligently submitting big data jobs for container clusters, relating to the field of big data processing technology. The method comprises receiving task configuration information, generating intermediate configuration data through field mapping and configuration standardization conversion, inputting the intermediate configuration data into a deep decision tree to obtain cluster resource status and task characteristics, determining the final submission method, generating task submission instructions based on a dynamic adaptation mechanism, executing tasks according to different submission methods, and monitoring task execution status in real time to optimize decision-making. The present invention improves the intelligent level of big data job submission, reduces resource waste, and improves task execution efficiency.

Claims

1. An intelligent submission method for big data jobs in container clusters, characterized by: include: Receive task configuration information submitted by users; This process constructs feature vectors for source and target fields, uses a preset word segmenter to generate word vectors, builds a field feature mapping table, and uses data from historical task records to calculate co-occurrence frequency weights. This process determines the initial mapping field set, extracts compute engine fields from the initial mapping field set, performs version configuration association analysis and migration rule generation, and performs version compatibility assessment. This completes configuration item standardization conversion and integrity verification, generating standard intermediate configuration data. Input standard intermediate configuration data into a deep decision tree, collect and analyze cluster resource status and load indicators through the computing resource evaluation node, parse the standard intermediate configuration data through the task feature analysis node, obtain task operation characteristics, and calculate resource evaluation scores, task feature scores, and historical task scores based on historical task data and intelligent routing strategies. Use the backpropagation algorithm to perform weighted fusion on multiple scores, determine the final submission method based on dynamic routing threshold comparison, and perform online update and optimization of corresponding model parameters based on actual execution results. Based on the template mapping mechanism, the corresponding template is selected according to the final submission method, and the template parameters are intelligently filled and optimized in combination with historical task data to generate task submission instructions; According to the task submission instruction, when the final submission mode is the first submission mode, a resource definition configuration file is generated and the task is submitted through a container cluster command; In the second submission mode, the command line tool that generates the computing framework submits the task; Monitor the execution status of tasks in real time, collect status information, resource information, and operation logs, and feed them back to the intelligent routing strategy to optimize the submission decisions of subsequent tasks.

2. The method according to claim 1, characterized in that This process constructs feature vectors for source and target fields, uses a preset word segmenter to generate word vectors, builds a field feature mapping table, and uses data from historical task records to calculate co-occurrence frequency weights. This process determines the initial mapping field set, extracts compute engine fields from the initial mapping field set, performs version configuration association analysis and migration rule generation, and performs version compatibility assessment. This completes the standardized conversion and integrity verification of configuration items, generating standard intermediate configuration data including: Perform feature vector construction on the source and target fields respectively. The feature vector construction uses a preset word segmenter to segment the field name, generates a basic word vector using the log-likelihood function of the center word and the context word, combines multi-dimensional semantic enhancement to generate word vectors, and extracts the field data type identifier and the boundary value of the value range to construct a field feature map. Based on the field feature mapping table, the mapping correlation between the source field and the target field is calculated. Field call data is extracted from historical task records to establish a field call relationship table. The co-occurrence frequency between fields is counted to obtain the call weight. The mapping correlation and call weight are weighted and combined to generate a field mapping score table. The target field with the highest score is selected as the initial mapping field set. Extract the calculation engine fields from the initial mapping field set, parse the version information and corresponding configuration items, and build a version configuration association table. Based on the version configuration association table, analyze the differences between configuration items in each version and generate a configuration migration rule set. Perform version compatibility assessment on the configuration items based on the configuration migration rule set to obtain a version-compatible configuration set. Perform standardized conversion of configuration items based on version-compatible configuration sets to generate standard configuration items; perform integrity verification on standard configuration items, complete configuration dependencies, and generate standard intermediate configuration data.

3. The method according to claim 2, characterized in that Use the preset word segmenter to segment the field name, use the log-likelihood function of the center word and the context word to generate the basic word vector, and combine it with multi-dimensional semantic enhancement to generate the word vector including: The word segmentation results are vectorized based on the central word and context words, the log-likelihood function between the central word and the word sequence in the corresponding context window is maximized, the conditional probability between the word vectors is calculated using the normalized exponent to generate a basic word vector; the character sequence of each word in the basic word vector is extracted, and the local character combination features are obtained using a sliding window scan to construct a character-level feature vector; Identifying word root and affix structures based on the character-level feature vector, combining corresponding word form change rules with the basic word vector to generate a word form vector; analyzing the syntactic dependency relationship of the word form vector to construct a syntax tree, extracting syntactic dependency features and fusing them with the word form vector to obtain a semantically enhanced vector; Extracting concept nodes and relationship edges from a preset dictionary, calculating the similarity between the semantic enhancement vector and the concept node to establish an entity mapping; extracting concept hierarchical relationships and attribute constraint information based on the entity mapping, and combining them with the semantic enhancement vector to obtain a knowledge association vector; Calculating a dynamic adjustment factor between the basic word vector and the knowledge association vector based on vector quality evaluation indicators, wherein the vector quality evaluation indicators include a vector clustering index, a vector discrimination index, and a vector coverage index; The basic word vector and the knowledge association vector are adaptively fused according to the dynamic adjustment factor to obtain a final word vector.

4. The method according to claim 1, wherein Input the standard intermediate configuration data into the deep decision tree, complete the collection and analysis of cluster resource status and load indicators through the computing resource evaluation node, parse the standard intermediate configuration data through the task feature analysis node, obtain the task operation characteristics, and calculate the resource evaluation score, task feature score, and historical task score based on historical task data and intelligent routing strategy. Use the backpropagation algorithm to perform weighted fusion on multiple scores, determine the final submission method based on dynamic routing threshold comparison, and update and optimize the corresponding model parameters online according to the actual execution results, including: Receive standard intermediate configuration data, generate a deep decision tree model, and construct resource evaluation nodes, task feature analysis nodes, and historical task analysis nodes; Cluster resource status indicator data is collected and normalized to obtain standardized resource indicators. This is then fed into a neural network trained based on historical resource data to generate a resource assessment score as the first scoring indicator. parsing task-related data from standard intermediate configuration data, obtaining task feature data through static analysis, constructing a task feature vector based on the task feature data, inputting the task feature vector into a pre-trained task classification model, and generating a task feature score as a second scoring indicator; Retrieving similar task records from a historical task database based on the task feature vector, extracting task execution data from the similar task records, and calculating and generating a historical task score as a third scoring indicator; The weight coefficient is calculated by the back propagation algorithm, and the first scoring index, the second scoring index and the third scoring index are weighted and integrated to generate a comprehensive routing score; Calculate a dynamic routing threshold based on cluster resource status and task distribution, compare the routing comprehensive score with the dynamic routing threshold, and determine the final submission method; The predicted deviation value between the final submission method and the actual execution result is calculated, and the neural network, task classification model and weight coefficient are updated online according to the predicted deviation value.

5. The method according to claim 4, characterized in that Generate resource assessment scores including: Convert the standardized resource indicators into a feature matrix, perform a convolution operation on the feature matrix to extract resource status features, input the resource status features into the fully connected layer, and map them through the sigmoid function to obtain the resource evaluation score; Generating task feature scores includes: The task feature vector is input into the attention layer to extract key features. The extracted key features are analyzed in time series based on the long short-term memory network. The probability distribution of different task types is calculated using the softmax function. The task feature score is obtained based on the weighted calculation of the probability distribution. Generate history task scores include: Based on the cosine similarity, the similarity between the task feature vector and the historical task record is calculated. According to the preset selection ratio, the historical task record with the highest similarity is selected, and the corresponding execution time, resource utilization and success rate are extracted. The historical task score is calculated through weighted average.

6. The method according to claim 1, characterized in that Based on the template mapping mechanism, the corresponding template is selected according to the final submission method. Combined with historical task data, the template parameters are intelligently filled and optimized to generate task submission instructions including: Establish a two-layer template mapping mechanism. The first layer selects the corresponding template type based on the final submission method. The second layer constructs a resource requirement index for the template type. The resource requirement index divides the task resource requirements into different gears, and each gear corresponds to a template parameter configuration scheme. Identify the critical path of parameter configuration from historical task data. The critical path is determined by analyzing the impact of parameter changes on task execution results. The parameter configuration combinations on the critical path are constructed into a parameter pattern library, where each parameter pattern contains a complete parameter dependency chain. Determining a basic template in the dual-layer template mapping mechanism according to the final submission method, matching a parameter configuration scheme in the resource requirement index based on the resource requirement of the current task, and selecting a parameter mode adapted to the parameter configuration scheme from the parameter mode library; The parameter pattern is applied to the basic template through iterative replacement, and parameter consistency check is performed after each replacement to finally generate a task submission instruction. After the task is executed, the task execution result is fed back to update the parameter pattern library.

7. The method according to claim 6, characterized in that The key paths to identify parameter configurations include: Construct a directed graph of task execution, where nodes are determined based on the execution phase, and edges between nodes are determined based on the data flow relationship between the execution phases; Marking configurable parameters on each node of the task execution directed graph, counting change records of the configurable parameters in historical task data, and calculating the degree of influence of the configurable parameters on adjacent execution phases; Iteratively calculating along the edges of the task execution directed graph using a message passing algorithm to obtain an impact propagation range of each configurable parameter, and calculating a cumulative impact value of the configurable parameter based on the impact propagation range; Sort the configurable parameters according to the cumulative impact values, select the parameter sequence with the largest cumulative impact value, and determine the critical path; Building a parameter pattern library, wherein each parameter pattern in the parameter pattern library contains a complete parameter dependency chain on the critical path; The critical path is converted into a parameter pattern and stored in the parameter pattern library. A parameter pattern score is established based on the usage frequency and the task success rate. Parameter patterns with scores lower than a preset score lower limit threshold are deleted, and the parameter pattern library is updated.

8. A big data job intelligent submission system for container clusters, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to receive task configuration information submitted by the user; The second unit is used to perform field mapping and configuration standardization conversion based on field feature similarity calculation and historical call relationships, combined with version compatibility assessment to generate standard intermediate configuration data; The third unit is used to input standard intermediate configuration data into the deep decision tree, obtain cluster resource status and load indicators through the computing resource evaluation node, obtain task operation characteristics through the task feature analysis node, and determine the final submission method based on intelligent routing strategy calculation; The fourth unit is used to select the corresponding template according to the final submission method based on the template mapping mechanism, and intelligently fill and optimize the template parameters in combination with historical task data to generate task submission instructions; A fifth unit is configured to generate a resource definition configuration file and submit the task through a container cluster command according to the task submission instruction when the final submission mode is the first submission mode; In the second submission mode, the command line tool that generates the computing framework submits the task; The sixth unit is used to monitor the execution status of tasks in real time, collect status information, resource information and operation logs, and feed them back to the intelligent routing strategy to optimize the submission decision of subsequent tasks.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images