A method for flexible task orchestration and dynamic adaptation for data governance

Abstract

The present invention discloses a method for flexible task scheduling and dynamic adaptation for data governance, comprising the following steps: S1: collecting task node historical records and training a model using the Lasso algorithm to analyze the impact of resource indicators on task duration; S2: expanding the task model, introducing new resource indicators and timeliness attributes; S3: building a resource model to monitor hardware devices and using the Raft algorithm to ensure data consistency; S4: pre-selecting and filtering nodes, sorting them based on the expected duration calculated by the model, and selecting the optimal node; S5: monitoring the health status of the task and adopting a retry strategy; S6: optimizing the model based on the actual running time. Through precise data collection and model training, combined with resource monitoring and task scheduling optimization, the present invention improves task execution efficiency and resource utilization, and ensures reliable task operation through a fault-tolerant mechanism.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and in particular to a method for flexible task orchestration and dynamic adaptation for data governance. Background Art

[0002] As enterprises accelerate their digital transformation, data volumes are growing exponentially, rapidly leaping from gigabytes to petabytes and beyond. Furthermore, data governance encompasses complex business scenarios such as metadata verification and data encryption, and resource requirements vary significantly across different tasks. For example, complex data analysis tasks rely heavily on CPU power, data warehouse updates place stringent demands on storage read and write performance, and real-time transaction data processing is extremely sensitive to timeliness.

[0003] Current data governance process orchestration often uses a graphical drag-and-drop component approach. While this simplifies process construction, the existing task scheduling center allocates resources based on device number order or evenly. This allocation model fails to perceive dynamic task resource requirements and changes in computing device load in real time, leading to an imbalance in resource allocation, resulting in idle device resources and inefficient task execution. This not only increases enterprise operating costs but also seriously affects the timeliness of data processing and the scientific nature of business decisions. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem in the existing technology that the resource requirements of the task cannot be accurately matched with the real-time status of the computing device, which leads to low task execution efficiency, serious waste of resources, insufficient response to the task, and inability to meet the efficient data governance needs of the enterprise. A method of flexible task orchestration and dynamic adaptation for data governance is proposed.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for flexible task orchestration and dynamic adaptation for data governance includes the following steps:

[0007] S1: Data collection and model training: Collect historical execution records of task nodes. The collected indicators include CPU usage, remaining CPU, total CPU, memory usage, remaining memory, total memory, storage rate, hard disk type, bandwidth usage, and task execution duration. Use the Lasso algorithm to analyze and process the collected data and train a calculation model that calculates the effect of CPU, memory, storage, hard disk type, and bandwidth on task execution duration.

[0008] S2: Task model expansion: expand CPU, memory, hard disk type, external network address resource indicators and task timeliness attributes;

[0009] S3: Resource Monitoring: Build a resource model to store the CPU, memory, storage, disk type, bandwidth usage, and maximum values ​​of each device metric for each task reported in real time by hardware devices. Use the Raft consensus algorithm to ensure strong data consistency across all nodes.

[0010] S4: Pre-selection, filtering, and optimal sorting: Based on the resource requirements of the nodes in S2 and the hardware resource status, anti-affinity, and resource reservation mechanisms in S3, all nodes that do not meet the requirements are filtered out. Based on the model data extended by S2, the calculation model of S1 is used on the filtered hardware nodes to calculate the expected runtime of each hardware node and prioritize them. Tasks are run on the optimal hardware nodes, and a data model is created to store the relationship between hardware devices and running tasks.

[0011] S5: Health Check and Fault Tolerance: This section establishes network communication with tasks running in S4, regularly monitors their health status, and records tasks whose runtimes far exceed the expected runtime in S4. It then applies different retry strategies based on the task's health status and the timeliness properties extended by S2.

[0012] S6: Model optimization: When the number of tasks whose actual running time recorded in S5 far exceeds the expected running time reaches a threshold, the Lasso model is retrained using recent historical data.

[0013] As a further technical solution of the present invention, S1 specifically includes:

[0014] S11: Create a task collection module to collect the resource usage and task execution time data of historical task execution;

[0015] S12: For missing values ​​in the data, the mean is used to fill in the missing values, and outliers are removed to ensure data quality. The preprocessed data is divided into training set and test set in a ratio of 7:3;

[0016] S13: The model is trained based on the Lasso algorithm, and key features are automatically screened by adjusting the regularization parameters. Finally, the model generalization ability is verified using a test set. The impact of various factors on task duration is analyzed based on the feature coefficients to form a calculation model.

[0017] As a further technical solution of the present invention, the S13 specifically includes:

[0018] S131: Define the loss function as:

[0019] ,

[0020] in: is the actual task duration of the i-th sample, is the transpose of the eigenvector of the i-th sample, is the feature coefficient vector, n is the number of training set samples, p is the number of features, is the regularization parameter; by adjusting the regularization parameter , so that the model minimizes the loss function on the training set. When it is larger, more characteristic coefficients will be Reduce to 0, thereby automatically screening out key features; when When it is smaller, the model retains more features;

[0021] S132: Determine the optimal regularization parameter using cross-validation method Specifically, the training set is divided into k subsets, one of which is selected as the validation set, and the rest are subsets as training subsets, for each candidate The model is trained with the same value and the mean square error is calculated on the validation set. The model with the smallest mean square error is selected. The value is taken as the optimal regularization parameter;

[0022] S133: Use the test set to verify the trained model and calculate the coefficient of determination between the model's predicted task duration and the actual task duration ,like If the value is greater than the set threshold (such as 0.8), the model is judged to have good generalization ability;

[0023] S134: According to the characteristic coefficient The size and positive and negative characteristics of the coefficients are used to analyze the impact of various factors on the task duration. Factors with larger absolute values ​​of characteristic coefficients have a greater impact on the task duration. A positive characteristic coefficient indicates that the factor will extend the task duration, and a negative characteristic coefficient indicates that the factor will shorten the task duration. Finally, these analysis results are integrated to form a calculation model.

[0024] As a further technical solution of the present invention, S2 specifically includes:

[0025] S21: For CPU type, define its performance index as , through the formula ,in is the CPU main frequency, The CPU architecture coefficient reflects the performance differences of different CPU architectures and converts the original CPU type into a standardized performance index.

[0026] S22: For memory type, define memory efficiency metrics ,in is the memory capacity, It is the memory read and write speed, reflecting the comprehensive performance of memory capacity and speed;

[0027] S23: For hard disk types, define hard disk efficiency indicators ,in is the write speed, For reading speed, It is the storage capacity, used to measure the overall performance of the hard disk;

[0028] S24: For external network address resource indicators, define network performance indicators ,in is the network bandwidth, The packet loss rate comprehensively reflects the transmission capacity and stability of the network;

[0029] S25: For the timeliness attribute of the task, define the timeliness index ,in is the task completion time, is the task start time, The task deadline is used to measure whether the task execution time is close to the deadline, thereby reflecting the timeliness of the task.

[0030] As a further technical solution of the present invention, S3 specifically includes:

[0031] S31: Design a standardized data structure to describe resource status, including core attributes such as device identification, indicator type (CPU / memory, etc.), real-time value, and timestamp to store device information;

[0032] S32: Obtains the CPU, memory, hard disk read / write speed, hard disk type, bandwidth, and latency resource usage data of multiple external network addresses in S2 for each task in real time on the hardware device side, as well as the maximum value of each device indicator, and packages and reports the data in a unified format.

[0033] S33: Deploy the Raft algorithm on the node that stores the hardware resource status, elect the master node, and establish a heartbeat detection mechanism between nodes to ensure the synchronization of multiple node states and strong consistency of node data.

[0034] As a further technical solution of the present invention, the S33 specifically includes:

[0035] S331: Master node election of Raft algorithm: Raft algorithm divides nodes into three states: Follower, Candidate and Leader. Initially, all nodes are in Follower state and define timeout parameters. The time interval for a node to transition from the Follower state to the Candidate state. If the Follower node does not receive the heartbeat packet from the Leader node within the specified time, an election is triggered and the node switches to the Candidate state. The Candidate node initiates an election request to other nodes and defines the voting parameters. , when the Candidate node obtains more than half of the votes ( is the total number of nodes), the Leader node is successfully elected;

[0036] S332: Heartbeat detection mechanism between nodes: The leader node periodically sends heartbeat packets to the follower node, defining the heartbeat interval parameters , if the Follower node is If no leader heartbeat packet is received within the specified time, the leader is considered invalid and the election process is restarted.

[0037] S333: Data Synchronization and Strong Consistency: Defining Log Item Sets , each log entry Contains index and term of office , the Leader node copies the new log to the Follower node, and through the majority commitment mechanism, ensures that the log is successfully copied on the majority of nodes before it is submitted, ensuring strong data consistency; define the majority parameters , when a log entry is at least If all nodes have successfully replicated the log, the log is considered committed.

[0038] As a further technical solution of the present invention, the S4 specifically includes:

[0039] S41: Build a data model to store operation records, including hardware device tags and running task identifiers;

[0040] S42: Process the CPU and memory of a single task in the device reported in S32. Calculate the hard real-time task based on the sum of running resources and reserved resources, reserve reasonable resources for the hard real-time task, and calculate the CPU and memory margin of the device.

[0041] S43: The CPU, memory, hard disk type, and external network address required by the task node in S2 are filtered against the device margin obtained in S42. If the task being run is a hard real-time task, the CPU and memory are calculated based on the sum of the required running resources and the reserved resources. Hardware devices that do not meet the requirements are filtered out. If no device is available, an email alert is triggered.

[0042] S44: Perform anti-affinity check on the hardware devices obtained in S43. When the number of hard real-time tasks running on a certain hardware device exceeds a certain threshold of the total number of hard real-time tasks, the node will not participate in the allocation of hard real-time tasks, thereby preventing the hard real-time tasks from being concentrated on the same hardware device.

[0043] S45: Calculate the estimated running time of the hardware devices that meet the requirements in S44 using the calculation model obtained in S13, and select the hardware device with the shortest running time as the task running carrier.

[0044] As a further technical solution of the present invention, the S5 specifically includes:

[0045] S51: Build a data model to store abnormal operation records, including task ID and abnormal number;

[0046] S52: Establish a network connection with the running task to regularly check its health status and actual running time. If the task running time exceeds a certain percentage of the estimated running time obtained in S45, it is saved as an exception record in the data model of S51;

[0047] S53: When the task status obtained from S52 is failed, the timeliness attribute extended from S2 is used to terminate the hard real-time task directly and mark it as failed. Soft real-time and batch tasks support three retries, with a different node selected for each retry.

[0048] As a further technical solution of the present invention, the S6 specifically includes:

[0049] S61: Periodically scan the data model of S51 to obtain the number of abnormal tasks;

[0050] S62: When the number of abnormalities of the task exceeds the threshold, the calculation model is retrained using the Lasso algorithm according to S1.

[0051] The beneficial effects of the present invention are:

[0052] 1. By collecting historical execution records of task nodes and using the Lasso algorithm to train the computing model, we can accurately analyze the execution of tasks in different hardware and network environments, providing a quantitative basis for task allocation. This changes the situation where traditional scheduling centers are unable to perceive the dynamic resource requirements of tasks in real time.

[0053] 2. During the task allocation process, comprehensive consideration is given to factors such as task resource requirements, hardware resource status, anti-affinity, and resource reservation mechanism, and pre-selection, filtering, and optimization sorting are performed to run tasks on the optimal hardware nodes, thereby avoiding resource allocation imbalance, reducing idle equipment resources, improving task execution efficiency and resource utilization, ensuring the supply of key task resources, and solving the problems of resource allocation imbalance and idle equipment resources in the current scheduling mechanism.

[0054] 3. Use heartbeat detection and fault-tolerant technology to monitor the task running status in real time. When a task encounters an abnormality, it can trigger a restart or migration operation in time based on the task's timeliness attributes, ensuring that the task can be responded to and processed in a timely manner, meeting the enterprise's requirements for timeliness in efficient data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flowchart of a method for flexible task orchestration and dynamic adaptation for data governance proposed by the present invention. DETAILED DESCRIPTION

[0056] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0057] Please see the attached Figure 1 A data governance-oriented task elastic orchestration and dynamic adaptation method includes the following steps:

[0058] S1: Data collection and model training: Collect historical execution records of task nodes. The collected indicators include CPU usage, remaining CPU, total CPU, memory usage, remaining memory, total memory, storage rate, hard disk type, bandwidth usage, and task execution duration. Use the Lasso algorithm to analyze and process the collected data and train a calculation model that calculates the effect of CPU, memory, storage, hard disk type, and bandwidth on task execution duration.

[0059] S11: Create a task collection module to collect the resource usage and task execution time data of historical task execution;

[0060] S12: For missing values ​​in the data, the mean is used to fill in the missing values, and outliers are removed to ensure data quality. The preprocessed data is divided into training set and test set in a ratio of 7:3;

[0061] S13: Based on the Lasso algorithm, the model is trained and key features are automatically selected by adjusting the regularization parameters. Finally, the generalization ability of the model is verified using the test set. The impact of various factors on task duration is analyzed based on the feature coefficients to form a calculation model.

[0062] S131: Define the loss function as:

[0063] ,

[0064] in: is the actual task duration of the i-th sample, is the transpose of the eigenvector of the i-th sample, is the feature coefficient vector, n is the number of training set samples, p is the number of features, is the regularization parameter; by adjusting the regularization parameter , so that the model minimizes the loss function on the training set. When it is larger, more characteristic coefficients will be Reduce to 0, thereby automatically screening out key features; when When it is smaller, the model retains more features;

[0065] S132: Determine the optimal regularization parameter using cross-validation method Specifically, the training set is divided into k subsets, one of which is selected as the validation set, and the rest are subsets as training subsets, for each candidate The model is trained with the same value and the mean square error is calculated on the validation set. The model with the smallest mean square error is selected. The value is taken as the optimal regularization parameter;

[0066] S133: Use the test set to verify the trained model and calculate the coefficient of determination between the model's predicted task duration and the actual task duration ,like If the value is greater than the set threshold (such as 0.8), the model is judged to have good generalization ability;

[0067] S134: According to the characteristic coefficient The size and positive and negative characteristics of the coefficients are used to analyze the impact of various factors on the task duration. Factors with larger absolute values ​​of characteristic coefficients have a greater impact on the task duration. A positive characteristic coefficient indicates that the factor will extend the task duration, and a negative characteristic coefficient indicates that the factor will shorten the task duration. Finally, these analysis results are integrated to form a calculation model.

[0068] S2: Task model expansion: expand CPU, memory, hard disk type, external network address resource indicators and task timeliness attributes;

[0069] S21: For CPU type, define its performance index as , through the formula ,in is the CPU main frequency, The CPU architecture coefficient reflects the performance differences of different CPU architectures and converts the original CPU type into a standardized performance index.

[0070] S22: For memory type, define memory efficiency metrics ,in is the memory capacity, It is the memory read and write speed, reflecting the comprehensive performance of memory capacity and speed;

[0071] S23: For hard disk types, define hard disk efficiency indicators ,in is the write speed, For reading speed, It is the storage capacity, used to measure the overall performance of the hard disk;

[0072] S24: For external network address resource indicators, define network performance indicators ,in is the network bandwidth, The packet loss rate comprehensively reflects the transmission capacity and stability of the network;

[0073] S25: For the timeliness attribute of the task, define the timeliness index ,in is the task completion time, is the task start time, The task deadline is used to measure whether the task execution time is close to the deadline, thereby reflecting the timeliness of the task.

[0074] S3: Resource Monitoring: Build a resource model to store the CPU, memory, storage, disk type, bandwidth usage, and maximum values ​​of each device metric for each task reported in real time by hardware devices. Use the Raft consensus algorithm to ensure strong data consistency across all nodes.

[0075] S31: Design a standardized data structure to describe resource status, including core attributes such as device identification, indicator type (CPU / memory, etc.), real-time value, and timestamp to store device information;

[0076] S32: Obtains the CPU, memory, hard disk read / write speed, hard disk type, bandwidth, and latency resource usage data of multiple external network addresses in S2 for each task in real time on the hardware device side, as well as the maximum value of each device indicator, and packages and reports the data in a unified format.

[0077] S33: Deploy the Raft algorithm on the node that stores the hardware resource status, elect a master node, and establish a heartbeat detection mechanism between nodes to ensure the synchronization of multiple node states and strong consistency of node data;

[0078] S331: Master node election of Raft algorithm: Raft algorithm divides nodes into three states: Follower, Candidate and Leader. Initially, all nodes are in Follower state and define timeout parameters. The time interval for a node to transition from the Follower state to the Candidate state. If the Follower node does not receive the heartbeat packet from the Leader node within the specified time, an election is triggered and the node switches to the Candidate state. The Candidate node initiates an election request to other nodes and defines the voting parameters. , when the Candidate node obtains more than half of the votes ( is the total number of nodes), the Leader node is successfully elected;

[0079] S332: Heartbeat detection mechanism between nodes: The leader node periodically sends heartbeat packets to the follower node, defining the heartbeat interval parameters , if the Follower node is If no leader heartbeat packet is received within the specified time, the leader is considered invalid and the election process is restarted.

[0080] S333: Data Synchronization and Strong Consistency: Defining Log Item Sets , each log entry Contains index and term of office , the Leader node copies the new log to the Follower node, and through the majority commitment mechanism, ensures that the log is successfully copied on the majority of nodes before it is submitted, ensuring strong data consistency; define the majority parameters , when a log entry is at least If all nodes have successfully replicated the log, the log is considered committed.

[0081] S4: Pre-selection, filtering, and optimal sorting: Based on the resource requirements of the nodes in S2 and the hardware resource status, anti-affinity, and resource reservation mechanisms in S3, all nodes that do not meet the requirements are filtered out. Based on the model data extended by S2, the calculation model of S1 is used on the filtered hardware nodes to calculate the expected runtime of each hardware node and prioritize them. Tasks are run on the optimal hardware nodes, and a data model is created to store the relationship between hardware devices and running tasks.

[0082] S41: Build a data model to store operation records, including hardware device tags and running task identifiers;

[0083] S42: Process the CPU and memory of a single task in the device reported in S32. Calculate the hard real-time task based on the sum of running resources and reserved resources, reserve reasonable resources for the hard real-time task, and calculate the CPU and memory margin of the device.

[0084] S43: The CPU, memory, hard disk type, and external network address required by the task node in S2 are filtered against the device margin obtained in S42. If the task being run is a hard real-time task, the CPU and memory are calculated based on the sum of the required running resources and the reserved resources. Hardware devices that do not meet the requirements are filtered out. If no device is available, an email alert is triggered.

[0085] S44: Perform anti-affinity check on the hardware devices obtained in S43. When the number of hard real-time tasks running on a certain hardware device exceeds a certain threshold of the total number of hard real-time tasks, the node will not participate in the allocation of hard real-time tasks, thereby preventing the hard real-time tasks from being concentrated on the same hardware device.

[0086] S45: Calculate the estimated running time of the hardware devices that meet the requirements in S44 using the calculation model obtained in S13, and select the hardware device with the shortest running time as the task running carrier.

[0087] S5: Health Check and Fault Tolerance: This section establishes network communication with tasks running in S4, regularly monitors their health status, and records tasks whose runtimes far exceed the expected runtime in S4. It then applies different retry strategies based on the task's health status and the timeliness properties extended by S2.

[0088] S51: Build a data model to store abnormal operation records, including task ID and abnormal number;

[0089] S52: Establish a network connection with the running task to regularly check its health status and actual running time. If the task running time exceeds a certain percentage of the estimated running time obtained in S45, it is saved as an exception record in the data model of S51;

[0090] S53: When the task status obtained from S52 is failed, the timeliness attribute extended from S2 is used to terminate the hard real-time task directly and mark it as failed. Soft real-time and batch tasks support three retries, with a different node selected for each retry.

[0091] S6: Model optimization: When the number of tasks whose actual running time recorded in S5 far exceeds the expected running time reaches a threshold, the Lasso model is retrained using recent historical data;

[0092] S61: Periodically scan the data model of S51 to obtain the number of abnormal tasks;

[0093] S62: When the number of abnormalities of the task exceeds the threshold, the calculation model is retrained using the Lasso algorithm according to S1.

[0094] Lasso regression algorithm: This algorithm introduces an L1 regularization term into the loss function to perform feature selection and parameter optimization based on CPU resource parameters, memory resource parameters, network environment parameters, disk read / write queue depth, average response time, disk type, etc. This algorithm is used to build a weight calculation model and evaluate task execution under different hardware and network environments.

[0095] Raft consensus algorithm: is an algorithm used to achieve consistency in distributed systems. It is mainly used to reach a consensus on the order of a series of operations among multiple nodes to ensure that the data in the distributed system remains consistent across all nodes. Even in the face of node failures, network delays, or message loss, it can guarantee the normal operation of the system and data consistency, thus providing a reliable foundation for distributed systems.

[0096] Heartbeat detection: This is a mechanism used to monitor node survival. Nodes periodically send specially crafted heartbeat packets to the dispatch center as a survival signal at set intervals (for example, in this solution, if a node fails to send a signal three times after a timeout (i.e., no signal within 1.5 seconds), it is considered a failure).

[0097] Task timeliness: Hard real-time tasks: These tasks must be completed within a fixed time. Soft real-time tasks: These tasks are prioritized but can be delayed. Batch tasks: These tasks can be executed during off-peak hours.

[0098] Example 1

[0099] The language used is Java. A data governance-oriented task elastic orchestration and dynamic adaptation method includes the following steps:

[0100] S1: Data collection and model training:

[0101] A task collection module was developed using Java backend. This module interacts with the existing data governance task scheduling system to collect historical execution records of task nodes from system logs and task execution records. The collected metrics include CPU usage, remaining capacity, total capacity; memory usage, remaining capacity, total capacity; storage speed, hard disk type, bandwidth usage, and task execution duration. During the data collection process, multi-threading technology was used to improve data collection efficiency, ensuring that large amounts of historical data could be quickly acquired. This data was then saved to the data storage module for subsequent analysis.

[0102] The collected data was preprocessed using the Java backend. First, code logic was written to remove data indicating task execution failures or task response times exceeding normal values. Next, the mean of various indicators was calculated for each task node. Missing values ​​were then filled with the mean to ensure data integrity and quality. After data cleaning and filling, the preprocessed data was divided into training and test sets in a 7:3 ratio to provide data support for model training and validation.

[0103] The Lasso algorithm is used to train the training set data, and key features are automatically selected by adjusting the regularization parameters. After training, the generalization ability of the model is verified using the test set, and the impact of various factors on task duration is analyzed based on the feature coefficients, ultimately forming a calculation model.

[0104] S2: Task model expansion: Based on the existing task model, resource indicators such as CPU, memory, hard disk type, and external network address are added. At the same time, the timeliness attribute of tasks is added, and the distinction between hard real-time tasks, soft real-time tasks, and batch tasks is used to provide a richer basis for subsequent task scheduling and resource allocation.

[0105] S3: Resource Monitoring:

[0106] Design a standardized JSON data structure that includes core attributes such as device ID, task ID, CPU usage, total amount, memory usage, total amount, storage speed, hard disk type, bandwidth usage, latency for different external network addresses, and timestamps. Develop a device resource collection module using Java backend. On the hardware device side, collect resource usage data for each task in real time, including CPU, memory, hard disk read / write speed, hard disk type, bandwidth, and latency for multiple external network addresses. Simultaneously, obtain the maximum value of each device metric. After collecting the data, encapsulate it in a unified JSON format and report it.

[0107] Drawing on the principles of the Raft consensus algorithm, a Java program with master node election, heartbeat detection, and data synchronization functions was developed. The election logic between nodes was implemented in the program. When a node received votes from more than half of the nodes within the specified time, it became the master node. A heartbeat detection mechanism was established between nodes, and the node sent heartbeat packets to other nodes at a set time interval (such as 1 second). If a node did not receive a heartbeat packet from a certain node within three consecutive heartbeat cycles, it was determined that the node might have failed, triggering a data synchronization operation. This mechanism ensures the status synchronization of multiple nodes, ensures strong consistency of node data, and provides accurate resource status information for task scheduling.

[0108] S4: Pre-selection, filtering and optimal sorting:

[0109] Constructing a data model in the data storage module to store the operation records of hardware device identification and running task identification;

[0110] When assigning tasks in the Java-developed task scheduling module, the CPU and memory headroom of devices for hard real-time tasks is calculated using 120% of the currently used resources, reserving appropriate resources for hard real-time tasks. Device headroom is screened based on the CPU, memory, hard disk type, and external network address required by the task node. If the running task is a hard real-time task, the CPU and memory headroom are calculated using 120% of the required capacity, filtering out hardware devices that do not meet the requirements. If no device is available, the Netty framework is used to trigger the email alert mechanism in the existing alert module through network communication, notifying relevant operations and maintenance personnel for timely processing.

[0111] The resource scheduling module then performs anti-affinity checks on the hardware devices that meet the requirements after screening. The data storage module obtains the number of hard real-time tasks running on each hardware device. If the number of hard real-time tasks on a hardware device exceeds 50% of the total number of hard real-time tasks, the node will not participate in the allocation of hard real-time tasks to prevent the concentration of hard real-time tasks on the same hardware device.

[0112] In the resource scheduling module, the estimated running time of hardware devices that meet the requirements after the anti-affinity check is calculated using the calculation model trained in S1. By inputting the resource indicators of the hardware devices into the calculation model, the estimated running time of the task on each hardware device is obtained. The hardware device with the shortest running time is selected as the carrier for task execution, achieving efficient task allocation.

[0113] S5: Health detection and fault tolerance guarantee:

[0114] Construct a data model in the data storage module to store abnormal operation records of task identification and abnormal number;

[0115] Use the Netty framework to establish a network connection with the running task and periodically check the task's health status and actual runtime. Set a scheduled task (e.g., check every 1 second) to obtain the task's runtime information and actual runtime through the network connection. If the task's runtime exceeds 50% of the estimated runtime calculated in S4, save it as an exception record in the abnormal runtime record data model of the data storage module.

[0116] Different strategies are adopted based on the health status and timeliness of the task. If the task status is failed and it is a hard real-time task, it is directly terminated and marked as failed. The Netty framework is used to trigger the email alert mechanism in the existing alert module through network communication to ensure timely notification of relevant personnel. For soft real-time tasks and batch tasks, three retries are supported, and a different node is selected for each retry. During the retry process, nodes are re-selected and allocated based on resource status and task requirements to increase the probability of successful task execution.

[0117] S6: Model optimization:

[0118] Use the Java backend to regularly scan the abnormal operation records in the data storage module to obtain the number of abnormal tasks. Set a scheduled task (at 1:00 a.m. every day) to obtain the number of abnormal tasks for each task from the abnormal operation records in the data storage module.

[0119] When the number of abnormalities in a task exceeds 100, the Lasso algorithm is used to retrain the computing model according to the steps of data collection and model training, and the recent historical task execution records are collected again. Data preprocessing, model training and verification are performed to enable the computing model to adapt to new task execution conditions and resource environment changes, thereby improving the accuracy of task allocation and scheduling.

[0120] From the above description, it can be seen that the above-mentioned embodiments of the present invention achieve the following technical effects: by collecting the historical execution records of task nodes, using the Lasso algorithm to train the calculation model, accurately analyzing the execution status of tasks in different hardware and network environments, providing a quantitative basis for task allocation, and changing the situation where traditional scheduling centers are unable to perceive the dynamic resource requirements of tasks in real time.

[0121] During the task allocation process, we comprehensively consider factors such as task resource requirements, hardware resource status, anti-affinity, and resource reservation mechanism, perform pre-selection filtering and optimal sorting, and run tasks on the optimal hardware nodes to avoid resource allocation imbalance, reduce idle equipment resources, improve task execution efficiency and resource utilization, ensure the supply of key task resources, and solve the problems of resource allocation imbalance and idle equipment resources in the current scheduling mechanism.

[0122] Use heartbeat detection and fault-tolerant technology to monitor the task running status in real time. When a task encounters an abnormality, it can trigger a restart or migration operation in time according to the task's timeliness attributes, ensuring that the task can be responded and processed in a timely manner, meeting the timeliness requirements of the enterprise's efficient data governance.

[0123] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention is limited to these examples. Within the scope of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0124] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the specification. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for flexible task orchestration and dynamic adaptation for data governance, characterized by: The following steps are involved: S1: Data collection and model training: Collect historical execution records of task nodes, use the Lasso algorithm to analyze and process the collected data, and train a calculation model for the effects of CPU, memory, storage, hard disk type, and bandwidth on task duration; S2: Task model expansion: expand CPU, memory, hard disk type, external network address resource indicators and task timeliness attributes; S3: Resource Monitoring: Build a resource model to store the CPU, memory, storage, disk type, bandwidth usage, and maximum values ​​of each device metric for each task reported in real time by hardware devices. Use the Raft consensus algorithm to ensure strong data consistency across all nodes. S4: Pre-selection, filtering, and optimal sorting: Based on the resource requirements of the nodes in S2 and the hardware resource status, anti-affinity, and resource reservation mechanisms in S3, all nodes that do not meet the requirements are filtered out. Based on the model data extended by S2, the calculation model of S1 is used on the filtered hardware nodes to calculate the expected runtime of each hardware node and prioritize them. Tasks are run on the optimal hardware nodes, and a data model is created to store the relationship between hardware devices and running tasks. S5: Health Check and Fault Tolerance: This section establishes network communication with tasks running in S4, regularly monitors their health status, and records tasks whose runtimes far exceed the expected runtime in S4. It then applies different retry strategies based on the task's health status and the timeliness properties extended by S2. S6: Model optimization: When the number of tasks whose actual running time recorded in S5 far exceeds the expected running time reaches a threshold, the Lasso model is retrained using recent historical data.

2. A data governance-oriented task elastic orchestration and dynamic adaptation method according to claim 1, characterized in that: Said S1 specifically includes: S11: Create a task collection module to collect the resource usage and task execution time data of historical task execution; S12: For missing values ​​in the data, the mean is used to fill in the missing values, and outliers are removed to ensure data quality. The preprocessed data is divided into training set and test set in a ratio of 7:3; S13: The model is trained based on the Lasso algorithm, and key features are automatically screened by adjusting the regularization parameters. Finally, the model generalization ability is verified using a test set. The impact of various factors on task duration is analyzed based on the feature coefficients to form a calculation model.

3. A data governance-oriented task elastic orchestration and dynamic adaptation method according to claim 2, characterized in that: The S13 specifically includes: S131: Define the loss function as: ， in: is the actual task duration of the i-th sample, is the transpose of the eigenvector of the i-th sample, is the feature coefficient vector, n is the number of training set samples, p is the number of features, is the regularization parameter; by adjusting the regularization parameter , so that the model minimizes the loss function on the training set. When it is larger, more characteristic coefficients will be Reduce to 0, thereby automatically screening out key features; when When it is smaller, the model retains more features; S132: Determine the optimal regularization parameter using cross-validation method Specifically, the training set is divided into k subsets, one of which is selected as the validation set, and the rest are subsets as training subsets, for each candidate The model is trained with the same value and the mean square error is calculated on the validation set. The model with the smallest mean square error is selected. The value is taken as the optimal regularization parameter; S133: Use the test set to verify the trained model and calculate the coefficient of determination between the model's predicted task duration and the actual task duration ,like If the value is greater than the set threshold, the model is judged to have good generalization ability; S134: According to the characteristic coefficient The size and positive and negative characteristics of the coefficients are used to analyze the impact of various factors on the task duration. Factors with larger absolute values ​​of characteristic coefficients have a greater impact on the task duration. A positive characteristic coefficient indicates that the factor will extend the task duration, and a negative characteristic coefficient indicates that the factor will shorten the task duration. Finally, these analysis results are integrated to form a calculation model.

4. A data governance-oriented task elastic orchestration and dynamic adaptation method according to claim 3, characterized in that: The S2 specifically includes: S21: For CPU type, define its performance index as , through the formula ,in is the CPU main frequency, The CPU architecture coefficient reflects the performance differences of different CPU architectures and converts the original CPU type into a standardized performance index. S22: For memory type, define memory efficiency metrics ,in is the memory capacity, It is the memory read and write speed, reflecting the comprehensive performance of memory capacity and speed; S23: For hard disk types, define hard disk efficiency indicators ,in For write speed, For reading speed, It is the storage capacity, used to measure the overall performance of the hard disk; S24: For external network address resource indicators, define network performance indicators ,in is the network bandwidth, The packet loss rate comprehensively reflects the transmission capacity and stability of the network; S25: For the timeliness attribute of the task, define the timeliness index ,in is the task completion time, is the task start time, The task deadline is used to measure whether the task execution time is close to the deadline, thereby reflecting the timeliness of the task.

5. A data governance-oriented task elastic orchestration and dynamic adaptation method according to claim 4, characterized in that: The S3 specifically includes: S31: Design a standardized data structure to describe resource status, including core attributes such as device identification, indicator type, real-time value, and timestamp to store device information; S32: Obtains the CPU, memory, hard disk read / write speed, hard disk type, bandwidth, and latency resource usage data of multiple external network addresses in S2 for each task in real time on the hardware device side, as well as the maximum value of each device indicator, and packages and reports the data in a unified format. S33: Deploy the Raft algorithm on the node that stores the hardware resource status, elect the master node, and establish a heartbeat detection mechanism between nodes to ensure the synchronization of multiple node states and strong consistency of node data.

6. A data governance-oriented task elastic orchestration and dynamic adaptation method according to claim 5, characterized in that: The S33 specifically includes: S331: Master node election of the Raft algorithm: The Raft algorithm divides nodes into three states: Follower, Candidate, and Leader. Initially, all nodes are in the Follower state. Define the timeout parameter. The time interval for a node to transition from the Follower state to the Candidate state. If the Follower node does not receive the heartbeat packet from the Leader node within the specified time, an election is triggered and the node switches to the Candidate state. The Candidate node initiates an election request to other nodes and defines the voting parameters. When the Candidate node obtains more than half of the votes, it is successfully elected as the Leader node; S332: Heartbeat detection mechanism between nodes: The leader node periodically sends heartbeat packets to the follower node, defining the heartbeat interval parameters , if the Follower node is If no leader heartbeat packet is received within the specified time, the leader is considered invalid and the election process is restarted. S333: Data Synchronization and Strong Consistency: Defining a Log Item Set , each log entry Contains index and term of office , the Leader node copies the new log to the Follower node, and through the majority commitment mechanism, ensures that the log is successfully copied on the majority of nodes before it is submitted, ensuring strong data consistency; define the majority parameters , is the total number of nodes. When a log entry is at least If all nodes have successfully replicated the log, the log is considered committed.

7. A data governance-oriented task elastic orchestration and dynamic adaptation method according to claim 6, characterized in that: The S4 specifically includes: S41: Build a data model to store operation records, including hardware device tags and running task identifiers; S42: Process the CPU and memory of a single task in the device reported in S32. Calculate the hard real-time task based on the sum of running resources and reserved resources, reserve reasonable resources for the hard real-time task, and calculate the CPU and memory margin of the device. S43: Based on the CPU performance index, memory efficiency index, hard disk type, and network performance index required by the task node in S2, the device margin obtained in S42 is filtered: only hardware devices whose CPU margin is not less than the task required CPU value, memory margin is not less than the task required memory value, hard disk type matches the task requirement, and network latency meets the task threshold are retained; if the running task is a hard real-time task, its total CPU and memory requirements are calculated as follows: the required running resources are the CPU computing power and memory capacity required by the task itself, and the reserved resources are the additional CPU redundancy and memory redundancy allocated to the task. The sum of the two is the total resource requirement of the hard real-time task. During the screening, only hardware devices with a device margin not less than the total resource requirement are retained; if no device is available, an email alert is triggered; S44: Perform anti-affinity check on the hardware devices obtained in S43. When the number of hard real-time tasks running on a certain hardware device exceeds a certain threshold of the total number of hard real-time tasks, the node does not participate in the allocation of hard real-time tasks. S45: Calculate the estimated running time of the hardware devices that meet the requirements in S44 using the calculation model obtained in S13, and select the hardware device with the shortest running time as the task running carrier.

8. The method for flexible task scheduling and dynamic adaptation for data governance according to claim 7 is characterized in that: The S5 specifically includes: S51: Build a data model to store abnormal operation records, including task ID and abnormal number; S52: Establish a network connection with the running task to regularly check its health status and actual running time. If the task running time exceeds a certain percentage of the estimated running time obtained in S45, it is saved as an exception record in the data model of S51; S53: When the task status obtained from S52 is failed, the timeliness attribute extended from S2 is used to terminate the hard real-time task directly and mark it as failed. Soft real-time and batch tasks support three retries, with a different node selected for each retry.

9. The method for flexible task scheduling and dynamic adaptation for data governance according to claim 8, characterized in that: The S6 specifically includes: S61: Periodically scan the data model of S51 to obtain the number of abnormal tasks; S62: When the number of abnormalities of the task exceeds the threshold, the calculation model is retrained using the Lasso algorithm according to S1.

Images