Runtime context parameter transfer and task instance dynamic scheduling method based on data flow

Abstract

The present invention provides a runtime context parameter transmission and task instance dynamic scheduling method based on data flow, which relates to the field of data processing technology, including receiving a task scheduling request, constructing a data flow dependency graph including task nodes and data flow direction relationships, and obtaining task instance runtime context information corresponding to each task node, including task instance identification, state information, and input and output parameter information. By analyzing the data flow dependency graph and runtime context information, a mapping relationship between input and output parameters of the task instance is established, and a parameter transmission rule is generated. The method monitors the task instance status information, and when the upstream task state changes, the upstream output parameter is passed to the downstream input parameter according to the parameter transmission rule, and when the downstream input parameter is complete, the dynamic scheduling execution of the downstream task instance is triggered, thereby realizing data flow-driven task scheduling and improving scheduling efficiency and flexibility.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a runtime context parameter transmission and task instance dynamic scheduling method based on data flow. Background Art

[0002] At present, as the scale and complexity of data continue to increase, data processing tasks are usually decomposed into multiple interdependent subtasks, forming a complex data flow. How to effectively manage and schedule these subtasks, ensure the correct transfer of data between different tasks, and achieve efficient execution of the entire data flow has become an important challenge.

[0003] Existing task scheduling methods mainly focus on static scheduling, that is, pre-defining the execution order and dependencies of tasks before they are executed. This method is difficult to adapt to dynamic changes in data flow, such as changes in the output data volume or data format of upstream tasks, and fluctuations in task execution time.

[0004] The prior art has the following defects and deficiencies:

[0005] 1. Difficulty in handling dynamic data flows: Static scheduling methods are difficult to adapt to dynamic changes in data flows, resulting in low efficiency in task execution or even task execution failure.

[0006] 2. Low parameter transfer efficiency: Existing parameter transfer mechanisms usually rely on middleware such as shared storage or message queues, which increases the overhead of data transmission and reduces the efficiency of task execution.

[0007] 3. Lack of flexible scheduling strategies: Existing scheduling methods usually lack flexible scheduling strategies and cannot dynamically adjust the execution order and resource allocation of tasks according to the actual execution status of tasks, resulting in low resource utilization. Summary of the invention

[0008] The embodiment of the present invention provides a runtime context parameter transmission and task instance dynamic scheduling method based on data flow, which can solve the problems in the prior art.

[0009] According to a first aspect of the embodiments of the present invention,

[0010] Provides runtime context parameter passing and task instance dynamic scheduling methods based on data flow, including:

[0011] Receive a task scheduling request, the task scheduling request includes task identification information and data flow configuration information, build a data flow dependency graph based on the data flow configuration information, the data flow dependency graph includes multiple task nodes and data flow direction relationships between task nodes, and for each task node in the data flow dependency graph, obtain corresponding task instance runtime context information, the task instance runtime context information includes task instance identification, task instance status information, task instance input parameter information, and task instance output parameter information;

[0012] Based on the data flow relationship between the task nodes in the data flow dependency graph, the runtime context information of the task instance is analyzed, a mapping relationship between the input parameter information of the task instance and the output parameter information of the task instance is established, and a parameter transfer rule is generated according to the mapping relationship, wherein the parameter transfer rule defines the transfer method of the output parameter information of the upstream task instance to the input parameter information of the downstream task instance;

[0013] Monitor the task instance status information of each task node in the data flow dependency graph. When it is detected that the status information of the upstream task instance has changed, the output parameter information of the upstream task instance is passed to the input parameter information of the downstream task instance according to the parameter passing rule. Based on the integrity of the input parameter information of the downstream task instance, it is judged whether the downstream task instance meets the execution condition. When the execution condition is met, a task instance scheduling instruction is sent to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance.

[0014] A data flow dependency graph is constructed based on the data flow configuration information, wherein the data flow dependency graph includes multiple task nodes and data flow relationships between the task nodes. For each task node in the data flow dependency graph, obtaining the corresponding task instance runtime context information includes:

[0015] The data flow configuration information includes task node information and data flow direction information, a node dependency matrix is ​​generated based on the data flow configuration information, the matrix elements in the node dependency matrix are used to characterize the data transfer relationship between task nodes, the task nodes are topologically sorted according to the node dependency matrix to obtain a task node sequence, and a data flow dependency graph is constructed based on the task node sequence;

[0016] For the task nodes in the data flow dependency graph, a context information model of the task instance is established, the context information model of the task instance includes task instance identification information, task instance running status information, task instance input parameter information and task instance output parameter information, a context acquisition function is generated based on the context information model of the task instance, the context acquisition function is used to obtain the context information of the task node at a specified timestamp, and the minimum update time interval of the context information is calculated according to the context acquisition function;

[0017] For each task node in the data flow dependency graph, a version vector of the task node is constructed, wherein the version vector includes version information of related task nodes that have a data transmission relationship with the task node, and a context information consistency check is performed based on the version vector. When the version information of the related task node meets a preset time deviation requirement, it is determined that the context information consistency check has passed, and the context information cache of the task node is updated according to the minimum update time interval. When a task node failure is detected, the context information of the task node is rolled back to a state corresponding to the most recent valid checkpoint.

[0018] For the task node in the data flow dependency graph, a context information model of the task instance is established, the context information model of the task instance includes task instance identification information, task instance running status information, task instance input parameter information and task instance output parameter information, a context acquisition function is generated based on the context information model of the task instance, the context acquisition function is used to obtain the context information of the task node at a specified timestamp, and the minimum update time interval of the context information is calculated according to the context acquisition function, including:

[0019] Constructing a context information model of a task instance, wherein the context information model includes instance identification information, task state information, input parameter information, and output parameter information; constructing a state transition matrix for the context information model, wherein the state transition matrix is ​​used to characterize the transition probability of task state information; calculating a distribution parameter of a state duration based on the state transition matrix; and evaluating a stability index of the task state according to the distribution parameter;

[0020] Constructing a context acquisition function based on the context information model, wherein the context acquisition function is used to obtain instance identification information, task status information, input parameter information, and output parameter information at a specified time point, analyze the dependency relationship between the input parameter information and the output parameter information, establish a parameter dependency relationship table, and calculate an integrity metric value of the context information, wherein the integrity metric value is a ratio of the number of valid input parameters to the total number of required input parameters;

[0021] Obtain the parameter change rate of the input parameter information and the output parameter information, establish an update threshold function based on the parameter change rate, the update threshold function includes an adjustment coefficient and a basic update time interval, calculate the minimum update time interval, construct a parameter timing correlation matrix, the parameter timing correlation matrix is ​​used to characterize the degree of timing correlation between parameters, establish a parameter prediction model based on the parameter timing correlation matrix, and calculate a prediction accuracy evaluation value according to the parameter prediction model.

[0022] Based on the data flow relationship between the task nodes in the data flow dependency graph, analyzing the runtime context information of the task instance, establishing a mapping relationship between the input parameter information of the task instance and the output parameter information of the task instance, and generating a parameter transfer rule according to the mapping relationship includes:

[0023] Construct a data flow dependency graph, wherein the data flow dependency graph includes a task node set, a data flow edge set and an edge weight matrix, wherein the edge weight matrix is ​​used to characterize the importance of the data flow edge, and a data flow metric value between any two task nodes is calculated based on the edge weight matrix, wherein the data flow metric value is related to the corresponding weight value in the edge weight matrix and the data flow between the nodes;

[0024] Establishing a context association matrix between task nodes, the context association matrix is ​​used to describe the degree of association between the output parameter set of the source task node and the input parameter set of the target task node, determining the correlation coefficient of any two task nodes based on the context association matrix, the correlation coefficient is calculated based on the parameter covariance of the task nodes and the parameter standard deviation of the task nodes, and determining a parameter mapping function based on the correlation coefficient, the parameter mapping function includes a mapping weight coefficient and a parameter conversion function;

[0025] A parameter dependency graph is constructed based on the parameter mapping function, the dependency distance between task nodes in the parameter dependency graph is calculated, the dependency strength between parameters is calculated based on the dependency distance, the dependency strength decays as the dependency distance increases, the transfer characteristics between parameter sequences with dependencies are analyzed, a parameter transfer rule template is generated based on the transfer characteristics, the parameter transfer rule template includes a source parameter set, a target parameter set, a trigger condition and a conversion action, and a priority is assigned to each parameter transfer rule based on the importance and execution frequency of the rule.

[0026] Constructing a parameter dependency graph based on the parameter mapping function, calculating the dependency distance between task nodes in the parameter dependency graph, calculating the dependency strength between parameters according to the dependency distance, wherein the dependency strength decays as the dependency distance increases, analyzing the transfer characteristics between parameter sequences with dependency relationships, and generating a parameter transfer rule template according to the transfer characteristics, including:

[0027] Based on the parameter dependency graph, the shortest path distance between task nodes is calculated, the shortest path distance is the minimum value of the sum of edge weights in the path connecting two task nodes, the type information of the task nodes is obtained and the semantic distance between the task nodes is calculated based on the similarity function of the parameter type, and the shortest path distance and the semantic distance are combined by a weight coefficient to obtain a comprehensive distance measurement value;

[0028] The dependency strength of the parameters is calculated according to the comprehensive distance metric value, and the dependency strength of the parameters is in the form of the product of a basic strength function and a timing influence factor, wherein the basic strength function decays exponentially with the increase of the comprehensive distance metric value, and the timing influence factor is related to the time interval. A parameter transfer chain is established based on the dependency strength of the parameters, and an integrity index and a reliability index of the parameter transfer chain are calculated, wherein the integrity index is the minimum value of the dependency strength of adjacent parameters in the parameter transfer chain, and the reliability index is calculated based on the error rate during the transfer process; a parameter transfer rule template is generated based on the parameter transfer chain, and the parameter transfer rule template includes a source mode, a target mode, a conversion rule, and a constraint condition.

[0029] The output parameter information of the upstream task instance is transmitted to the input parameter information of the downstream task instance according to the parameter transmission rule, and whether the downstream task instance meets the execution condition is determined based on the integrity of the input parameter information of the downstream task instance. When the execution condition is met, a task instance scheduling instruction is sent to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance, including:

[0030] Obtain an output parameter set of an upstream task instance, calculate the accuracy and timeliness index of each parameter in the output parameter set, determine a parameter quality evaluation value based on the accuracy and timeliness index, and convert the parameter quality evaluation value into a parameter availability index according to a preset weight coefficient;

[0031] Counting the input parameter set required by the downstream task instance, calculating the parameter integrity metric value of the output parameter set of the upstream task instance and the input parameter set required by the downstream task instance, the parameter integrity metric value being the ratio of the number of parameters actually received to the number of parameters required for the task, and calculating the condition satisfaction based on the parameter integrity metric value and the parameter availability index;

[0032] Obtain resource consumption information and task urgency information of downstream task instances, calculate the basic task priority based on the resource consumption information and the task urgency information, obtain the load balancing factor at the current time point, the load balancing factor decreases as the current load of the system increases, and multiply the basic task priority by the load balancing factor to obtain the dynamic task priority;

[0033] Based on the dynamic priority of the task, a matching degree evaluation is performed on the available computing resources, and the matching degree evaluation includes a matching degree calculation of the resource feature dimension. A scheduling instruction including a task identifier, a resource identifier, priority information, execution constraints and parameter requirements is generated, and the scheduling instruction is verified for resource availability and parameter completeness.

[0034] A second aspect of an embodiment of the present invention provides a runtime context parameter transmission and task instance dynamic scheduling system based on data flow, including:

[0035] A first unit is used to receive a task scheduling request, wherein the task scheduling request includes task identification information and data flow configuration information, construct a data flow dependency graph based on the data flow configuration information, wherein the data flow dependency graph includes multiple task nodes and data flow direction relationships between task nodes, and for each task node in the data flow dependency graph, obtain corresponding task instance runtime context information, wherein the task instance runtime context information includes task instance identification, task instance status information, task instance input parameter information, and task instance output parameter information;

[0036] The second unit is used to analyze the runtime context information of the task instance based on the data flow relationship between the task nodes in the data flow dependency graph, establish a mapping relationship between the input parameter information of the task instance and the output parameter information of the task instance, and generate a parameter transfer rule according to the mapping relationship, wherein the parameter transfer rule defines the transfer method of the output parameter information of the upstream task instance to the input parameter information of the downstream task instance;

[0037] The third unit is used to monitor the task instance status information of each task node in the data flow dependency graph. When it is detected that the status information of the upstream task instance changes, the output parameter information of the upstream task instance is passed to the input parameter information of the downstream task instance according to the parameter passing rule. Based on the integrity of the input parameter information of the downstream task instance, it is judged whether the downstream task instance meets the execution condition. When the execution condition is met, the task instance scheduling instruction is sent to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance.

[0038] A third aspect of the embodiments of the present invention

[0039] An electronic device is provided, comprising:

[0040] processor;

[0041] a memory for storing processor-executable instructions;

[0042] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0043] A fourth aspect of the embodiments of the present invention is:

[0044] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0045] The beneficial effects of this application are as follows:

[0046] 1. Realize dynamic scheduling of task instances: By monitoring the status changes and dependencies of task instances, downstream task execution is automatically triggered without manual intervention, which improves the efficiency and automation of task scheduling.

[0047] 2. Flexible parameter passing: Based on the data flow dependency graph and runtime context information, parameter mapping relationships and passing rules are established to achieve automatic passing of upstream task output parameters to downstream task input parameters, avoiding the tedious operation of manual parameter configuration and reducing human errors.

[0048] 3. Improve data stream processing efficiency: Through parameter passing rules and dynamic scheduling mechanism, ensure that downstream tasks are executed only when the input parameters are complete, avoid invalid scheduling and waiting, and improve the overall efficiency of data stream processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a process of a method for dynamically scheduling task instances and contextual parameters based on data streams at runtime according to an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of parameter definition based on task context in a business process according to an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of dynamically generating a sub-instance based on parameter values ​​passed in during runtime context according to an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of scheduling, allocating resources, and executing a sub-example of an embodiment of the present invention through distributed scheduling;

[0053] Figure 5 A schematic diagram of a process for dynamically scheduling a task instance based on runtime context parameter transmission of a data stream according to an embodiment of the present invention;

[0054] Figure 6 The present invention is a schematic diagram of the structure of a runtime context parameter transmission and task instance dynamic scheduling system based on data flow according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0057] refer to Figures 1 to 5 The runtime context parameter transmission and task instance dynamic scheduling method based on data flow in the embodiment of the present invention includes:

[0058] S101. Receive a task scheduling request, the task scheduling request includes task identification information and data flow configuration information, build a data flow dependency graph based on the data flow configuration information, the data flow dependency graph includes multiple task nodes and data flow relationships between task nodes, and for each task node in the data flow dependency graph, obtain the corresponding task instance runtime context information, the task instance runtime context information includes task instance identification, task instance status information, task instance input parameter information and task instance output parameter information;

[0059] S102. Based on the data flow relationship between the task nodes in the data flow dependency graph, analyze the runtime context information of the task instance, establish a mapping relationship between the input parameter information of the task instance and the output parameter information of the task instance, and generate a parameter transfer rule according to the mapping relationship, wherein the parameter transfer rule defines the transfer method of the output parameter information of the upstream task instance to the input parameter information of the downstream task instance;

[0060] S103. Monitor the task instance status information of each task node in the data flow dependency graph. When it is detected that the status information of the upstream task instance has changed, the output parameter information of the upstream task instance is passed to the input parameter information of the downstream task instance according to the parameter passing rule. Based on the integrity of the input parameter information of the downstream task instance, it is determined whether the downstream task instance meets the execution condition. When the execution condition is met, a task instance scheduling instruction is sent to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance.

[0061] In an optional implementation, a data flow dependency graph is constructed based on the data flow configuration information, the data flow dependency graph includes multiple task nodes and data flow relationships between task nodes, and for each task node in the data flow dependency graph, obtaining the corresponding task instance runtime context information includes:

[0062] The data flow configuration information includes task node information and data flow direction information, a node dependency matrix is ​​generated based on the data flow configuration information, the matrix elements in the node dependency matrix are used to characterize the data transfer relationship between task nodes, the task nodes are topologically sorted according to the node dependency matrix to obtain a task node sequence, and a data flow dependency graph is constructed based on the task node sequence;

[0063] For the task nodes in the data flow dependency graph, a context information model of the task instance is established, the context information model of the task instance includes task instance identification information, task instance running status information, task instance input parameter information and task instance output parameter information, a context acquisition function is generated based on the context information model of the task instance, the context acquisition function is used to obtain the context information of the task node at a specified timestamp, and the minimum update time interval of the context information is calculated according to the context acquisition function;

[0064] For each task node in the data flow dependency graph, a version vector of the task node is constructed, wherein the version vector includes version information of related task nodes that have a data transmission relationship with the task node, and a context information consistency check is performed based on the version vector. When the version information of the related task node meets a preset time deviation requirement, it is determined that the context information consistency check has passed, and the context information cache of the task node is updated according to the minimum update time interval. When a task node failure is detected, the context information of the task node is rolled back to a state corresponding to the most recent valid checkpoint.

[0065] The data flow dependency graph construction and context information management method is used to construct the dependency graph of data flow tasks and manage the runtime context information of task instances to ensure data consistency and task reliability.

[0066] First, configure the data flow information. The data flow configuration information is stored in JSON format and contains task node information and data flow information. Task node information includes task ID, task name, task type, etc. Data flow information describes the data transfer relationship between task nodes, for example, the output of task A is the input of task B.

[0067] Then, a node dependency matrix is ​​generated. A node dependency matrix is ​​generated based on the data flow information in the data flow configuration information. The rows and columns of the matrix represent task nodes, and the matrix elements represent the data transfer relationship between task nodes. If the output of task A is the input of task B, the element value at the corresponding position in the matrix is ​​1, otherwise it is 0.

[0068] Next, perform topological sorting. According to the node dependency matrix, perform topological sorting on the task nodes to obtain a task node sequence. Topological sorting ensures that the order of task execution complies with the data dependency. For example, in the above example, the task node sequence is task_A ->task_B ->task_C.

[0069] Then, a data flow dependency graph is constructed. Based on the topologically sorted task node sequence, a data flow dependency graph is constructed. The data flow dependency graph shows the dependency relationship between task nodes in the form of a directed acyclic graph. The nodes in the graph represent tasks, and the edges represent data flows.

[0070] Establish a task instance context information model. Establish a task instance context information model for each task node. The model includes task instance identification information (such as task instance ID), task instance running status information (such as running, successful, failed), task instance input parameter information, and task instance output parameter information.

[0071] Generate a context acquisition function. Generate a context acquisition function based on the context information model of the task instance. The context acquisition function is used to obtain the context information of the task node at a specified timestamp. Calculate the minimum update time interval of the context information. According to the context acquisition function, calculate the minimum update time interval of the context information. This time interval is used to control the frequency of updating the context information to avoid performance waste caused by frequent updates. For example, the minimum update time interval can be determined based on the execution frequency of the task or the frequency of data changes.

[0072] Build a version vector for the task node. Build a version vector for each task node. The version vector contains the version information of the related task nodes that have a data transfer relationship with the task node. The version information can be a timestamp or a version number.

[0073] Perform a context information consistency check. Perform a context information consistency check based on the version vector. When the version information of the relevant task node meets the preset time deviation requirement, the context information consistency check is determined to have passed. For example, if task B depends on task A, and the difference between the version timestamps of task A and task B does not exceed 1 minute, the context information consistency check is considered to have passed.

[0074] Update the task node's context information cache. Update the task node's context information cache regularly according to the minimum update interval.

[0075] Task node failure processing: When a task node failure is detected, the context information of the task node is rolled back to the state corresponding to the most recent valid checkpoint to ensure data consistency.

[0076] This application can achieve:

[0077] Improve data consistency: Ensure that the context information of the task node is consistent through version vector and context information consistency checks to avoid data conflicts and errors. Enhance task reliability: Roll back the context information of the task node to the most recent valid checkpoint through the task node failure handling mechanism to ensure reliable execution of the task. Optimize resource utilization: Control the frequency of context information updates through the minimum update time interval to avoid resource waste caused by frequent updates and improve system efficiency.

[0078] In an optional implementation, for a task node in the data flow dependency graph, a context information model of a task instance is established, the context information model of the task instance includes task instance identification information, task instance running status information, task instance input parameter information and task instance output parameter information, a context acquisition function is generated based on the context information model of the task instance, the context acquisition function is used to acquire context information of the task node at a specified timestamp, and the minimum update time interval of the context information is calculated according to the context acquisition function, including:

[0079] Constructing a context information model of a task instance, wherein the context information model includes instance identification information, task state information, input parameter information, and output parameter information; constructing a state transition matrix for the context information model, wherein the state transition matrix is ​​used to characterize the transition probability of task state information; calculating a distribution parameter of a state duration based on the state transition matrix; and evaluating a stability index of the task state according to the distribution parameter;

[0080] Constructing a context acquisition function based on the context information model, wherein the context acquisition function is used to obtain instance identification information, task status information, input parameter information, and output parameter information at a specified time point, analyze the dependency relationship between the input parameter information and the output parameter information, establish a parameter dependency relationship table, and calculate an integrity metric value of the context information, wherein the integrity metric value is a ratio of the number of valid input parameters to the total number of required input parameters;

[0081] Obtain the parameter change rate of the input parameter information and the output parameter information, establish an update threshold function based on the parameter change rate, the update threshold function includes an adjustment coefficient and a basic update time interval, calculate the minimum update time interval, construct a parameter timing correlation matrix, the parameter timing correlation matrix is ​​used to characterize the degree of timing correlation between parameters, establish a parameter prediction model based on the parameter timing correlation matrix, and calculate a prediction accuracy evaluation value according to the parameter prediction model.

[0082] For the task nodes in the data flow dependency graph, this embodiment provides a method for constructing a task instance context information model and calculating the minimum update time interval to improve the efficiency of task monitoring and management.

[0083] First, construct the context information model of the task instance. The model contains four key information: instance identification information, task running status information, input parameter information, and output parameter information. For example, for a task instance named "data cleaning", its identification information can be UUID, such as "a1b2c3d4-e5f6-7890-1234-567890abcdef"; the running status information can be "running", "successful", "failed", etc.; the input parameter information can be the data file name and cleaning rules to be cleaned; the output parameter information can be the data file name and cleaning report after cleaning.

[0084] Next, a state transition matrix is ​​constructed for the context information model. This matrix is ​​used to characterize the migration probability of task state information. For example, the "Running" state may migrate to the "Success" or "Failure" state, while the "Success" state usually does not migrate to other states. By counting a large number of task instance state transition data, the state transition probability can be obtained, for example, the probability of going from "Running" to "Success" is 90%, and the probability of going to "Failure" is 10%. Assume that the state transition probability of the "Data Cleaning" task instance is as follows: the probability of going from "Running" to "Success" is 90%, and the probability of going to "Failure" is 10%; the "Success" and "Failure" states remain unchanged.

[0085] Then, the distribution parameters of the state duration are calculated based on the state transition matrix. For example, the average duration, maximum duration, minimum duration, etc. of each state can be calculated. These parameters can be used to evaluate the stability of the task state. For example, if the average duration of the "Running" state is long, it may indicate that the task execution efficiency is low. Assuming that the average duration of the "Running" state of the "Data Cleaning" task instance is 10 minutes, the duration of the "Success" and "Failure" states can be considered infinite.

[0086] Afterwards, a context acquisition function is constructed based on the context information model. The function is used to obtain instance identification information, task status information, input parameter information, and output parameter information at a specified time point. For example, this information can be obtained by querying a database or calling an API interface.

[0087] Next, the dependency between the input parameter information and the output parameter information is analyzed, and a parameter dependency table is established. For example, the name of the cleaned data file depends on the name of the data file to be cleaned and the cleaning rules. Then, the integrity metric of the context information is calculated. The integrity metric is defined as the ratio of the number of valid input parameters to the total number of required input parameters. For example, if the "data cleaning" task requires two input parameters: the data file name and the cleaning rules, but only the data file name is actually provided, the integrity metric is 50%.

[0088] Then, the parameter change rate of the input parameter information and the output parameter information is obtained. For example, the number of changes or the magnitude of changes of each parameter within a period of time can be calculated.

[0089] Create an update threshold function based on the parameter change rate. This function contains an adjustment coefficient and a basic update interval. For example, the adjustment coefficient can be set to 2 and the basic update interval can be set to 1 minute. Then, when the parameter change rate exceeds a certain threshold, the update interval will be shortened to 0.5 minutes. Assuming that the input parameter "data file name" of the "data cleaning" task changes once a minute, the update interval is 0.5 minutes.

[0090] Finally, the minimum update time interval is calculated. This time interval is used to determine when the context information needs to be updated. For example, if the minimum update time interval is 1 minute, the context acquisition function needs to be called every 1 minute to obtain the latest context information. And construct a parameter time series correlation matrix, which is used to characterize the degree of time series correlation between parameters. For example, the correlation coefficient of two parameters in the time series can be calculated. Then, a parameter prediction model is established based on the parameter time series correlation matrix. For example, a time series analysis method can be used to predict the future value of the parameter. Finally, the prediction accuracy evaluation value is calculated based on the parameter prediction model. For example, the mean square error can be used to evaluate the prediction accuracy. Assuming that the output parameter "the amount of data after cleaning" of the "data cleaning" task is highly correlated with the input parameter "the amount of data to be cleaned", a simple linear regression model can be established to predict the "amount of data after cleaning".

[0091] This application can achieve:

[0092] Improve task monitoring efficiency: By obtaining real-time context information of task instances, task anomalies can be discovered and handled in a timely manner, improving task monitoring efficiency. Optimize resource scheduling: By analyzing the distribution parameters of task state duration, the stability and resource requirements of tasks can be evaluated, thereby optimizing resource scheduling strategies. Improve prediction accuracy: By establishing a parameter prediction model, the future behavior of tasks can be predicted, such as predicting the completion time or resource consumption of tasks, thereby improving prediction accuracy.

[0093] In an optional implementation, based on the data flow relationship between the task nodes in the data flow dependency graph, analyzing the runtime context information of the task instance, establishing a mapping relationship between the task instance input parameter information and the task instance output parameter information, and generating parameter transfer rules according to the mapping relationship include:

[0094] Construct a data flow dependency graph, wherein the data flow dependency graph includes a task node set, a data flow edge set and an edge weight matrix, wherein the edge weight matrix is ​​used to characterize the importance of the data flow edge, and a data flow metric value between any two task nodes is calculated based on the edge weight matrix, wherein the data flow metric value is related to the corresponding weight value in the edge weight matrix and the data flow between the nodes;

[0095] Establishing a context association matrix between task nodes, the context association matrix is ​​used to describe the degree of association between the output parameter set of the source task node and the input parameter set of the target task node, determining the correlation coefficient of any two task nodes based on the context association matrix, the correlation coefficient is calculated based on the parameter covariance of the task nodes and the parameter standard deviation of the task nodes, and determining a parameter mapping function based on the correlation coefficient, the parameter mapping function includes a mapping weight coefficient and a parameter conversion function;

[0096] A parameter dependency graph is constructed based on the parameter mapping function, the dependency distance between task nodes in the parameter dependency graph is calculated, the dependency strength between parameters is calculated based on the dependency distance, the dependency strength decays as the dependency distance increases, the transfer characteristics between parameter sequences with dependencies are analyzed, a parameter transfer rule template is generated based on the transfer characteristics, the parameter transfer rule template includes a source parameter set, a target parameter set, a trigger condition and a conversion action, and a priority is assigned to each parameter transfer rule based on the importance and execution frequency of the rule.

[0097] In order to effectively manage and transfer parameter information in complex workflows, this embodiment provides a parameter transfer rule generation method based on data flow dependency and context information.

[0098] First, construct a data flow dependency graph. The graph contains a set of nodes representing each task and a set of directed edges representing the data flow between tasks. Each edge is assigned a weight to indicate the importance of the data flow. For example, in a video processing flow, the weight of the edge from the "video decoding" task to the "video encoding" task can be set to 0.9, indicating that the decoding result is crucial to the encoding process, while the weight of the edge from the "subtitle addition" task to the "video encoding" task can be set to 0.5, indicating that the subtitle information has a relatively small impact on the encoding process. By analyzing the weights of the edges and the data flow between the nodes, the data flow metric between any two task nodes can be calculated. For example, if the "video decoding" task outputs 10MB of data to the "video encoding" task, the data flow metric between the two tasks is 0.9 * 10 = 9.

[0099] Next, a context association matrix is ​​established between task nodes. This matrix describes the degree of association between the output parameter set of the source task node and the input parameter set of the target task node. For example, the output parameters of the "video decoding" task include resolution, frame rate, and encoding format, while the input parameters of the "video encoding" task include resolution, frame rate, and bit rate. By analyzing the relationship between these parameters, the correlation coefficient of the two task nodes can be determined. For example, if the resolution and frame rate are highly correlated, their correlation coefficient is close to 1, while the correlation between the encoding format and bit rate is low, the coefficient is close to 0. According to the correlation coefficient, the parameter mapping function is determined. The function contains a mapping weight coefficient and a parameter conversion function. For example, if the output resolution of the source task is 1920x1080 and the target task requires a resolution of 720x480, the parameter conversion function can perform the corresponding scaling operation, and the mapping weight coefficient can be set according to the importance of the resolution.

[0100] Then, a parameter dependency graph is constructed based on the parameter mapping function. The dependency distance between the task nodes in the graph is calculated. For example, if there is no other task between the "video decoding" task and the "video encoding" task, their dependency distance is 1. If there is also a "watermarking" task in the middle, the dependency distance is 2. The dependency strength between the parameters is calculated based on the dependency distance, and the dependency strength decays as the dependency distance increases. For example, the dependency strength is 1 when the distance is 1, and the dependency strength is 0.5 when the distance is 2. The transfer characteristics between parameter sequences with dependencies are analyzed. For example, the resolution parameter needs to be consistent throughout the video processing flow. A parameter transfer rule template is generated based on the transfer characteristics. The template contains a source parameter set, a target parameter set, a trigger condition, and a conversion action. For example, a rule template can be: when the "video decoding" task is completed, the decoded resolution is passed to the "video encoding" task, and the corresponding scaling operation is performed.

[0101] Finally, each parameter passing rule is assigned a priority based on its importance and execution frequency. For example, the rule for ensuring video resolution consistency has a higher priority than the rule for adding subtitles. Assuming that in the video processing flow, three parameters, resolution, frame rate, and bit rate, need to be passed, they are assigned priorities 1, 2, and 3 respectively based on their importance and frequency of use.

[0102] This application can achieve:

[0103] Improve parameter transfer efficiency. By analyzing the dependencies between tasks and the correlation between parameters, parameter transfer rules are automatically generated, avoiding the tedious operation of manually configuring parameters, thereby improving the efficiency of parameter transfer. Reduce parameter transfer errors. By establishing a parameter dependency graph and a context association matrix, errors in the parameter transfer process, such as parameter type mismatch, missing parameter values, etc., can be effectively identified and avoided, thereby improving the accuracy of parameter transfer. Enhance the flexibility of workflows. Through the parameter transfer rule template, the parameter transfer rules can be dynamically adjusted according to different task requirements, thereby enhancing the flexibility and adaptability of the workflow, for example, different encoding parameters can be selected according to different video formats.

[0104] In an optional implementation, a parameter dependency graph is constructed based on the parameter mapping function, a dependency distance between task nodes in the parameter dependency graph is calculated, a dependency strength between parameters is calculated based on the dependency distance, the dependency strength decays as the dependency distance increases, and a transfer characteristic between parameter sequences with dependencies is analyzed. Generating a parameter transfer rule template based on the transfer characteristic includes:

[0105] Based on the parameter dependency graph, the shortest path distance between task nodes is calculated, the shortest path distance is the minimum value of the sum of edge weights in the path connecting two task nodes, the type information of the task nodes is obtained and the semantic distance between the task nodes is calculated based on the similarity function of the parameter type, and the shortest path distance and the semantic distance are combined by a weight coefficient to obtain a comprehensive distance measurement value;

[0106] The dependency strength of the parameters is calculated according to the comprehensive distance metric value, and the dependency strength of the parameters is in the form of the product of a basic strength function and a timing influence factor, wherein the basic strength function decays exponentially with the increase of the comprehensive distance metric value, and the timing influence factor is related to the time interval. A parameter transfer chain is established based on the dependency strength of the parameters, and an integrity index and a reliability index of the parameter transfer chain are calculated, wherein the integrity index is the minimum value of the dependency strength of adjacent parameters in the parameter transfer chain, and the reliability index is calculated based on the error rate during the transfer process; a parameter transfer rule template is generated based on the parameter transfer chain, and the parameter transfer rule template includes a source mode, a target mode, a conversion rule, and a constraint condition.

[0107] The parameter dependency graph construction and transfer rule template generation method generates a parameter transfer rule template by analyzing the dependency relationship between parameters to improve the efficiency and accuracy of parameter transfer.

[0108] First, construct a parameter dependency graph. Consider each parameter as a node in the graph. If there is a dependency between two parameters, an edge is established between the two nodes. The weight of the edge represents the strength of the dependency, which can be determined by domain expert scoring or data-driven methods. For example, if parameter A and parameter B jointly affect the result of a task, the weight of the edge can be set according to their influence on the result. Assuming that the influence of parameter A is 0.8 and the influence of parameter B is 0.6, the weight of the edge between them can be set to 0.8 * 0.6 = 0.48.

[0109] Next, calculate the dependency distance between task nodes. Dependency distance is used to measure the strength of the dependency relationship between parameters. First, calculate the shortest path distance between task nodes, that is, the minimum value of the sum of edge weights in the path connecting two task nodes. For example, the shortest path from parameter A to parameter C passes through parameter B, and the AB edge weight is 0.48 and the BC edge weight is 0.7, then the shortest path distance of AC is 0.48 + 0.7 = 1.18. At the same time, obtain the type information of the task node, such as the data type and value range of the parameter, and calculate the semantic distance between the task nodes. The semantic distance is calculated based on the similarity function of the parameter type. For example, if the data types of two parameters are the same, their semantic distance is small. Assuming that the data type similarity of parameter A and parameter C is 0.9, their semantic distance can be set to 1 - 0.9 = 0.1. Then, the shortest path distance and semantic distance are combined through the weight coefficient to obtain a comprehensive distance metric. For example, if the weight of the shortest path distance is set to 0.6 and the weight of the semantic distance is set to 0.4, the comprehensive distance metric of AC is 1.18 * 0.6 + 0.1 * 0.4 = 0.748.

[0110] Then, the dependency strength of the parameters is calculated based on the comprehensive distance metric. The dependency strength of the parameters is in the form of the product of the basic strength function and the time series impact factor. The basic strength function decays exponentially as the comprehensive distance metric increases. For example, exp(-comprehensive distance metric) can be used as the basic strength function. The time series impact factor is related to the time interval. For example, if the time interval between two parameters is long, the dependency strength between them will decrease. Assuming that the time interval between parameters A and C is 1 day, the time series impact factor can be set to 0.9. Assuming that the comprehensive distance metric of AC is 0.748, the dependency strength of AC is exp(-0.748) * 0.9 = 0.428.

[0111] Based on the dependency strength of the parameters, a parameter transfer chain is established. The parameter transfer chain represents the order and dependency of parameter transfer. For example, if parameter A depends on parameter B, and parameter B depends on parameter C, a parameter transfer chain C -> B -> A can be established. The integrity index and reliability index of the parameter transfer chain are calculated. The integrity index is the minimum value of the dependency strength of adjacent parameters in the parameter transfer chain. The reliability index is calculated based on the error rate in the transfer process. For example, the error rate of parameter transfer can be calculated based on historical data. Assuming that the dependency strength of CB is 0.6 and the dependency strength of BA is 0.428, the integrity index of the transfer chain is 0.428. Assuming that the error rate in the transfer process is 0.1, the reliability index of the transfer chain is 1 - 0.1 = 0.9.

[0112] Finally, a parameter transfer rule template is generated based on the parameter transfer chain. The parameter transfer rule template contains source mode, target mode, conversion rules and constraints. The source mode represents the starting state of parameter transfer, the target mode represents the target state of parameter transfer, the conversion rule describes how to convert the parameter from the source mode to the target mode, and the constraint condition limits the conditions of parameter transfer. For example, if the value range of parameter C is [0, 1] and the value range of parameter B is [1, 2], then a conversion rule can be set: B = C + 1, and a constraint condition: C>= 0.

[0113] This application can achieve:

[0114] Improve parameter transfer efficiency: By analyzing the dependency relationship between parameters, unnecessary parameter transfer can be avoided, thereby improving the efficiency of parameter transfer. Improve parameter transfer accuracy: By considering the type information and time interval of the parameters, the dependency strength of the parameters can be calculated more accurately, thereby improving the accuracy of parameter transfer. Enhance the interpretability of parameter transfer rules: The parameter transfer rule template contains the source mode, target mode, conversion rules and constraints, which can clearly describe the logic of parameter transfer, thereby enhancing the interpretability of parameter transfer rules.

[0115] In an optional implementation, the output parameter information of the upstream task instance is transferred to the input parameter information of the downstream task instance according to the parameter transfer rule, and whether the downstream task instance meets the execution condition is determined based on the integrity of the input parameter information of the downstream task instance. When the execution condition is met, a task instance scheduling instruction is sent to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance, which includes:

[0116] Obtain an output parameter set of an upstream task instance, calculate the accuracy and timeliness index of each parameter in the output parameter set, determine a parameter quality evaluation value based on the accuracy and timeliness index, and convert the parameter quality evaluation value into a parameter availability index according to a preset weight coefficient;

[0117] Counting the input parameter set required by the downstream task instance, calculating the parameter integrity metric value of the output parameter set of the upstream task instance and the input parameter set required by the downstream task instance, the parameter integrity metric value being the ratio of the number of parameters actually received to the number of parameters required for the task, and calculating the condition satisfaction based on the parameter integrity metric value and the parameter availability index;

[0118] Obtain resource consumption information and task urgency information of downstream task instances, calculate the basic task priority based on the resource consumption information and the task urgency information, obtain the load balancing factor at the current time point, the load balancing factor decreases as the current load of the system increases, and multiply the basic task priority by the load balancing factor to obtain the dynamic task priority;

[0119] Based on the dynamic priority of the task, a matching degree evaluation is performed on the available computing resources, and the matching degree evaluation includes a matching degree calculation of the resource feature dimension. A scheduling instruction including a task identifier, a resource identifier, priority information, execution constraints and parameter requirements is generated, and the scheduling instruction is verified for resource availability and parameter completeness.

[0120] A task dynamic scheduling method based on parameter transfer and condition judgment is used to efficiently execute tasks in a distributed computing environment. The core of this method is to dynamically determine whether the downstream task meets the execution conditions based on the output parameter information of the upstream task, and perform priority scheduling based on the system load, thereby improving resource utilization and task execution efficiency.

[0121] First, obtain the output parameter set of the upstream task instance. For example, the output parameters of an image recognition task may include the type of object identified, the confidence level, and the image feature vector. Then, calculate the accuracy and timeliness indicators of each output parameter separately. For example, the confidence level can be used as an accuracy indicator, and the image acquisition time can be used as a timeliness indicator. Assume that the identified object type is "cat", the confidence level is 95%, and the image acquisition time is 1 minute ago.

[0122] Next, the parameter quality assessment value is determined based on the accuracy and timeliness indicators. For example, the parameter quality assessment value can be obtained by weighting and summing the confidence and timeliness indicators according to the preset rules. Assuming the confidence weight is 0.8 and the timeliness weight is 0.2, the parameter quality assessment value is 0.8 * 95% + 0.2 * (100% - 1 / 60) ≈ 76.3%. Then, the parameter quality assessment value is converted into the parameter availability indicator according to the preset weight coefficient. Assuming the weight coefficient is 0.9, the parameter availability indicator is 0.9 * 76.3% ≈ 68.7%.

[0123] After that, the input parameter set required by the downstream task instance is counted. For example, a target tracking task requires the input of the object type, image feature vector, and target initial position. Then, the parameter completeness metric of the output parameter set of the upstream task instance and the input parameter set required by the downstream task instance is calculated. For example, if the upstream task provides the object type and image feature vector, and the target initial position is provided by other tasks, the parameter completeness metric is 2 / 3 ≈ 66.7%. Next, the condition satisfaction is calculated based on the parameter completeness metric and the parameter availability index. For example, the condition satisfaction can be obtained by weighted summing the two. Assuming that the parameter completeness metric weight is 0.7 and the parameter availability index weight is 0.3, the condition satisfaction is 0.7 * 66.7% + 0.3 * 68.7% ≈ 67.2%.

[0124] If the condition satisfaction reaches a preset threshold, such as 70%, proceed to the next step. Obtain resource consumption information and task urgency information of the downstream task instance. For example, resource consumption information may include the required number of CPU cores, memory size, and GPU type, while task urgency information can be evaluated based on the deadline and importance of the task. Assume that the target tracking task requires 2 CPU cores, 4GB of memory, and an NVIDIA Tesla T4 GPU, and the task urgency is "high".

[0125] The basic task priority is calculated based on the resource consumption information and the task urgency information. For example, the basic task priority can be obtained by weighting the resource consumption and the urgency according to the preset rules. Assuming that the resource consumption weight is 0.6 and the urgency weight is 0.4, the basic task priority is 0.6 * (2 / 8 + 4 / 16 + 1 / 1) + 0.4 * 1 = 0.85, where the total resources of the CPU, memory, and GPU are 8 cores, 16GB, and 1, respectively.

[0126] Get the load balancing factor at the current time point. The load balancing factor decreases as the current system load increases. Assuming that the current system load is high, the load balancing factor is 0.5. Multiply the task basic priority by the load balancing factor to get the task dynamic priority. The task dynamic priority is 0.85 * 0.5 = 0.425.

[0127] The matching degree of available computing resources is evaluated based on the dynamic priority of the task. The matching degree evaluation includes the matching degree calculation of the resource feature dimension. For example, matching is performed based on the GPU type required for the task and the type of available GPU. A scheduling instruction containing task identification, resource identification, priority information, execution constraints, and parameter requirements is generated. For example, the scheduling instruction may contain the ID of the target tracking task, the ID of the assigned GPU, the priority of 0.425, the execution time limit of 1 hour, and the required input parameter information. The scheduling instruction is verified for resource availability and parameter completeness. After ensuring that the allocated resources are available and the parameters are complete, the task instance scheduling instruction is sent to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance.

[0128] This application can achieve:

[0129] Improve resource utilization: By dynamically scheduling according to task requirements and system load, computing resources can be effectively utilized to avoid resource waste and bottlenecks. Improve task execution efficiency: Through priority scheduling and parameter prediction mechanisms, urgent and important tasks can be executed first, and task waiting time can be reduced, thereby improving overall task execution efficiency. Enhance system stability: Through the load balancing mechanism, system overload can be avoided and the stable operation of the system can be ensured.

[0130] Figure 6 FIG. 1 is a schematic diagram of a structure of a runtime context parameter transmission and task instance dynamic scheduling system based on data stream according to an embodiment of the present invention. Figure 6 As shown, the system comprises:

[0131] A first unit is used to receive a task scheduling request, wherein the task scheduling request includes task identification information and data flow configuration information, construct a data flow dependency graph based on the data flow configuration information, wherein the data flow dependency graph includes multiple task nodes and data flow direction relationships between task nodes, and for each task node in the data flow dependency graph, obtain corresponding task instance runtime context information, wherein the task instance runtime context information includes task instance identification, task instance status information, task instance input parameter information, and task instance output parameter information;

[0132] The second unit is used to analyze the runtime context information of the task instance based on the data flow relationship between the task nodes in the data flow dependency graph, establish a mapping relationship between the input parameter information of the task instance and the output parameter information of the task instance, and generate a parameter transfer rule according to the mapping relationship, wherein the parameter transfer rule defines the transfer method of the output parameter information of the upstream task instance to the input parameter information of the downstream task instance;

[0133] The third unit is used to monitor the task instance status information of each task node in the data flow dependency graph. When it is detected that the status information of the upstream task instance changes, the output parameter information of the upstream task instance is passed to the input parameter information of the downstream task instance according to the parameter passing rule. Based on the integrity of the input parameter information of the downstream task instance, it is judged whether the downstream task instance meets the execution condition. When the execution condition is met, the task instance scheduling instruction is sent to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance.

[0134] According to a third aspect of the embodiments of the present invention,

[0135] An electronic device is provided, comprising:

[0136] processor;

[0137] a memory for storing processor-executable instructions;

[0138] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0139] A fourth aspect of the embodiments of the present invention is:

[0140] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0141] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A runtime context parameter transmission and task instance dynamic scheduling method based on data flow, characterized in that: include: Receive a task scheduling request, the task scheduling request includes task identification information and data flow configuration information, build a data flow dependency graph based on the data flow configuration information, the data flow dependency graph includes multiple task nodes and data flow direction relationships between task nodes, and for each task node in the data flow dependency graph, obtain corresponding task instance runtime context information, the task instance runtime context information includes task instance identification, task instance status information, task instance input parameter information, and task instance output parameter information; Based on the data flow relationship between the task nodes in the data flow dependency graph, the runtime context information of the task instance is analyzed, a mapping relationship between the input parameter information of the task instance and the output parameter information of the task instance is established, and a parameter transfer rule is generated according to the mapping relationship, wherein the parameter transfer rule defines the transfer method of the output parameter information of the upstream task instance to the input parameter information of the downstream task instance; Monitor the task instance status information of each task node in the data flow dependency graph, and when it is detected that the status information of the upstream task instance changes, pass the output parameter information of the upstream task instance to the input parameter information of the downstream task instance according to the parameter transfer rule, and judge whether the downstream task instance meets the execution condition based on the integrity of the input parameter information of the downstream task instance. When the execution condition is met, send a task instance scheduling instruction to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance; A data flow dependency graph is constructed based on the data flow configuration information, wherein the data flow dependency graph includes a plurality of task nodes and data flow directions between the task nodes. For each task node in the data flow dependency graph, obtaining the corresponding task instance runtime context information includes: The data flow configuration information includes task node information and data flow direction information, a node dependency matrix is ​​generated based on the data flow configuration information, the matrix elements in the node dependency matrix are used to characterize the data transfer relationship between task nodes, the task nodes are topologically sorted according to the node dependency matrix to obtain a task node sequence, and a data flow dependency graph is constructed based on the task node sequence; For the task nodes in the data flow dependency graph, a context information model of the task instance is established, the context information model of the task instance includes task instance identification information, task instance running status information, task instance input parameter information and task instance output parameter information, a context acquisition function is generated based on the context information model of the task instance, the context acquisition function is used to obtain the context information of the task node at a specified timestamp, and the minimum update time interval of the context information is calculated according to the context acquisition function; For each task node in the data flow dependency graph, a version vector of the task node is constructed, wherein the version vector includes version information of related task nodes that have a data transmission relationship with the task node, and a context information consistency check is performed based on the version vector. When the version information of the related task node meets a preset time deviation requirement, it is determined that the context information consistency check has passed, and the context information cache of the task node is updated according to the minimum update time interval. When a task node failure is detected, the context information of the task node is rolled back to a state corresponding to the most recent valid checkpoint.

2. The method according to claim 1, characterized in that For the task node in the data flow dependency graph, a context information model of the task instance is established, the context information model of the task instance includes task instance identification information, task instance running status information, task instance input parameter information and task instance output parameter information, a context acquisition function is generated based on the context information model of the task instance, the context acquisition function is used to obtain the context information of the task node at a specified timestamp, and the minimum update time interval of the context information is calculated according to the context acquisition function, including: Constructing a context information model of a task instance, wherein the context information model includes instance identification information, task state information, input parameter information, and output parameter information; constructing a state transition matrix for the context information model, wherein the state transition matrix is ​​used to characterize the transition probability of task state information; calculating a distribution parameter of a state duration based on the state transition matrix; and evaluating a stability index of the task state according to the distribution parameter; Constructing a context acquisition function based on the context information model, wherein the context acquisition function is used to obtain instance identification information, task status information, input parameter information, and output parameter information at a specified time point, analyze the dependency relationship between the input parameter information and the output parameter information, establish a parameter dependency relationship table, and calculate an integrity metric value of the context information, wherein the integrity metric value is a ratio of the number of valid input parameters to the total number of required input parameters; Obtain the parameter change rate of the input parameter information and the output parameter information, establish an update threshold function based on the parameter change rate, the update threshold function includes an adjustment coefficient and a basic update time interval, calculate the minimum update time interval, construct a parameter timing correlation matrix, the parameter timing correlation matrix is ​​used to characterize the degree of timing correlation between parameters, establish a parameter prediction model based on the parameter timing correlation matrix, and calculate a prediction accuracy evaluation value according to the parameter prediction model.

3. The method according to claim 1, characterized in that Based on the data flow relationship between the task nodes in the data flow dependency graph, analyzing the runtime context information of the task instance, establishing a mapping relationship between the input parameter information of the task instance and the output parameter information of the task instance, and generating a parameter transfer rule according to the mapping relationship includes: Construct a data flow dependency graph, wherein the data flow dependency graph includes a task node set, a data flow edge set and an edge weight matrix, wherein the edge weight matrix is ​​used to characterize the importance of the data flow edge, and a data flow metric value between any two task nodes is calculated based on the edge weight matrix, wherein the data flow metric value is related to the corresponding weight value in the edge weight matrix and the data flow between the nodes; Establishing a context association matrix between task nodes, the context association matrix is ​​used to describe the degree of association between the output parameter set of the source task node and the input parameter set of the target task node, determining the correlation coefficient of any two task nodes based on the context association matrix, the correlation coefficient is calculated based on the parameter covariance of the task nodes and the parameter standard deviation of the task nodes, and determining a parameter mapping function based on the correlation coefficient, the parameter mapping function includes a mapping weight coefficient and a parameter conversion function; A parameter dependency graph is constructed based on the parameter mapping function, the dependency distance between task nodes in the parameter dependency graph is calculated, the dependency strength between parameters is calculated based on the dependency distance, the dependency strength decays as the dependency distance increases, the transfer characteristics between parameter sequences with dependencies are analyzed, a parameter transfer rule template is generated based on the transfer characteristics, the parameter transfer rule template includes a source parameter set, a target parameter set, a trigger condition and a conversion action, and a priority is assigned to each parameter transfer rule based on the importance and execution frequency of the rule.

4. The method according to claim 3, characterized in that Constructing a parameter dependency graph based on the parameter mapping function, calculating the dependency distance between task nodes in the parameter dependency graph, calculating the dependency strength between parameters according to the dependency distance, wherein the dependency strength decays as the dependency distance increases, analyzing the transfer characteristics between parameter sequences with dependency relationships, and generating a parameter transfer rule template according to the transfer characteristics, including: Based on the parameter dependency graph, the shortest path distance between task nodes is calculated, the shortest path distance is the minimum value of the sum of edge weights in the path connecting two task nodes, the type information of the task nodes is obtained and the semantic distance between the task nodes is calculated based on the similarity function of the parameter type, and the shortest path distance and the semantic distance are combined by a weight coefficient to obtain a comprehensive distance measurement value; The dependency strength of the parameters is calculated according to the comprehensive distance metric value, and the dependency strength of the parameters is in the form of the product of a basic strength function and a timing influence factor, wherein the basic strength function decays exponentially with the increase of the comprehensive distance metric value, and the timing influence factor is related to the time interval. A parameter transfer chain is established based on the dependency strength of the parameters, and an integrity index and a reliability index of the parameter transfer chain are calculated, wherein the integrity index is the minimum value of the dependency strength of adjacent parameters in the parameter transfer chain, and the reliability index is calculated based on the error rate during the transfer process; a parameter transfer rule template is generated based on the parameter transfer chain, and the parameter transfer rule template includes a source mode, a target mode, a conversion rule, and a constraint condition.

5. The method according to claim 1, characterized in that The output parameter information of the upstream task instance is transmitted to the input parameter information of the downstream task instance according to the parameter transmission rule, and whether the downstream task instance meets the execution condition is determined based on the integrity of the input parameter information of the downstream task instance. When the execution condition is met, a task instance scheduling instruction is sent to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance, including: Obtain an output parameter set of an upstream task instance, calculate the accuracy and timeliness index of each parameter in the output parameter set, determine a parameter quality evaluation value based on the accuracy and timeliness index, and convert the parameter quality evaluation value into a parameter availability index according to a preset weight coefficient; Counting the input parameter set required by the downstream task instance, calculating the parameter integrity metric value of the output parameter set of the upstream task instance and the input parameter set required by the downstream task instance, the parameter integrity metric value being the ratio of the number of parameters actually received to the number of parameters required for the task, and calculating the condition satisfaction based on the parameter integrity metric value and the parameter availability index; Obtain resource consumption information and task urgency information of downstream task instances, calculate the basic task priority based on the resource consumption information and the task urgency information, obtain the load balancing factor at the current time point, the load balancing factor decreases as the current load of the system increases, and multiply the basic task priority by the load balancing factor to obtain the dynamic task priority; Based on the dynamic priority of the task, a matching degree evaluation is performed on the available computing resources, and the matching degree evaluation includes a matching degree calculation of the resource feature dimension. A scheduling instruction including a task identifier, a resource identifier, priority information, execution constraints and parameter requirements is generated, and the scheduling instruction is verified for resource availability and parameter completeness.

6. A runtime context parameter transmission and task instance dynamic scheduling system based on data flow, used to implement the method as described in any one of claims 1 to 5, characterized in that: include: A first unit is used to receive a task scheduling request, wherein the task scheduling request includes task identification information and data flow configuration information, construct a data flow dependency graph based on the data flow configuration information, wherein the data flow dependency graph includes multiple task nodes and data flow direction relationships between task nodes, and for each task node in the data flow dependency graph, obtain corresponding task instance runtime context information, wherein the task instance runtime context information includes task instance identification, task instance status information, task instance input parameter information, and task instance output parameter information; The second unit is used to analyze the runtime context information of the task instance based on the data flow relationship between the task nodes in the data flow dependency graph, establish a mapping relationship between the input parameter information of the task instance and the output parameter information of the task instance, and generate a parameter transfer rule according to the mapping relationship, wherein the parameter transfer rule defines the transfer method of the output parameter information of the upstream task instance to the input parameter information of the downstream task instance; The third unit is used to monitor the task instance status information of each task node in the data flow dependency graph. When it is detected that the status information of the upstream task instance changes, the output parameter information of the upstream task instance is passed to the input parameter information of the downstream task instance according to the parameter passing rule. Based on the integrity of the input parameter information of the downstream task instance, it is judged whether the downstream task instance meets the execution condition. When the execution condition is met, the task instance scheduling instruction is sent to the task scheduling system to trigger the dynamic scheduling execution of the downstream task instance.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images