A multi-line collaborative scheduling method and device based on artificial intelligence

Abstract

The application provides a multi-line cooperative scheduling method and device based on artificial intelligence, which comprises the following steps: acquiring a resource scheduling topology graph and receiving a task execution request; according to task demand parameters carried in the task execution request, analyzing a resource demand type and an execution constraint condition corresponding to the task execution request; starting from a starting level node of the resource scheduling topology graph, traversing all level nodes in the resource scheduling topology graph, and acquiring a resource prediction model or a task allocation model associated with a current traversed level node; based on the resource demand type and the execution constraint condition, generating a resource allocation instruction corresponding to the current traversed level node through the resource prediction model or the task allocation model associated with the current traversed level node; and according to the resource allocation instruction, dynamically allocating a computing task corresponding to the task execution request to a target resource node, and triggering the target resource node to execute the computing task. The application can improve resource allocation accuracy and system robustness.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a multi-line collaborative scheduling method and device based on artificial intelligence. Background Technology

[0002] In AI middleware platforms, multi-line collaboration is required to optimize the efficiency of computational task execution and resource utilization through dynamic resource allocation. Existing scheduling methods typically employ static resource pool partitioning strategies or task allocation based on a single decision model (such as relying solely on resource load indicators or task priorities), treating resource prediction and task allocation as independent processes during scheduling. However, in existing methods, the resource prediction model cannot capture the fluctuation characteristics of dynamic resource occupancy in real time (such as load peaks caused by sudden tasks), resulting in prediction results lagging behind the actual resource status. The task allocation model, lacking prior resource capacity assessment, may allocate high-priority tasks to nodes that are about to reach full capacity, causing task execution interruptions or resource contention conflicts. At the same time, the separation of resource prediction and task allocation in traditional methods prevents the formation of a closed-loop feedback loop, making it impossible for the scheduling strategy to be dynamically optimized based on real-time task execution data, leading to increased resource idle rates and a heightened risk of task deadline defaults. Summary of the Invention

[0003] This invention provides a multi-line collaborative scheduling method and device based on artificial intelligence.

[0004] In a first aspect, embodiments of the present invention provide a multi-line collaborative scheduling method based on artificial intelligence. The method includes: acquiring a resource scheduling topology map and receiving a task execution request; each hierarchical node of the resource scheduling topology map is associated with a resource prediction model or a task allocation model, and in each execution path of the resource scheduling topology map, the hierarchical node associated with the task allocation model is located after the hierarchical node associated with the resource prediction model; wherein, the resource prediction model is a resource prediction model trained based on a resource occupancy feature set, and the task allocation model is a task allocation model generated based on a task priority feature set; and according to the task requirements carried in the task execution request... The parameters are parsed to determine the resource requirement type and execution constraints corresponding to the task execution request. Starting from the initial level node of the resource scheduling topology, all level nodes in the resource scheduling topology are traversed to obtain the resource prediction model or task allocation model associated with the currently traversed level node. Based on the resource requirement type and execution constraints, a resource allocation instruction corresponding to the currently traversed level node is generated through the resource prediction model or task allocation model associated with the currently traversed level node. According to the resource allocation instruction, the computing task corresponding to the task execution request is dynamically allocated to the target resource node, and the target resource node is triggered to execute the computing task.

[0005] Secondly, embodiments of the present invention provide a multi-line collaborative scheduling device, comprising: a memory storing a computer program; and a processor for loading the computer program to implement the artificial intelligence-based multi-line collaborative scheduling method as described above.

[0006] The multi-line collaborative scheduling method based on artificial intelligence provided by this invention obtains a resource scheduling topology map and parses the resource demand type and execution constraints in the task execution request. Starting from the initial level node of the topology map, it traverses all level nodes of the associated resource prediction model or task allocation model, and generates resource allocation instructions based on dynamic resource prediction and priority-driven allocation strategies. Among them, the resource prediction model predicts the available resource capacity of the target resource node by analyzing the dynamic occupancy feature set of the historical resource dataset, and the task allocation model generates task allocation weights by extracting task deadlines, dependencies and resource exclusivity parameters from the task priority feature set, and dynamically allocates the computation tasks to the target resource node in combination with the execution constraints. In this way, the resource prediction model can accurately predict the dynamic load trend of resource nodes based on time-series characteristics, avoiding allocation failures caused by resource fluctuations. The task allocation model adjusts task scheduling weights through multi-dimensional priority parameters, ensuring that high-urgency tasks receive priority resource supply. By sequentially linking resource prediction and task allocation in the hierarchical nodes of the topology graph, a "prediction-decision" closed loop is formed, enabling real-time coordination between resource capacity assessment and task priority matching. This achieves dual optimization of resource utilization efficiency and task execution timeliness in complex multi-task scenarios. Simultaneously, incremental training of the resource prediction model and task allocation model is performed based on real-time collected task execution data, continuously optimizing the adaptability of the topology graph's scheduling strategy and further improving resource allocation accuracy and system robustness. In addition, the conflict resolution model assesses the impact of resource competition tasks and dynamically arbitrates conflicts, effectively reducing task queuing delays and resource preemption costs, ensuring efficient collaborative execution of multi-task tasks. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a flowchart of a multi-line collaborative scheduling method based on artificial intelligence provided in an embodiment of the present invention.

[0009] Figure 2 This is a schematic diagram of the composition of a multi-line collaborative scheduling device provided in an embodiment of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] Please see Figure 1 , Figure 1 A flowchart of an AI-based multi-line collaborative scheduling method is provided as an embodiment of the present invention. This AI-based multi-line collaborative scheduling method can be executed by a multi-line collaborative scheduling device and may include the following steps:

[0012] This invention provides a multi-line cooperative scheduling method based on artificial intelligence, which includes the following steps:

[0013] Step S100: Obtain the resource scheduling topology and receive task execution requests; each level node of the resource scheduling topology is associated with a resource prediction model or a task allocation model, and in each execution path of the resource scheduling topology, the level node associated with the task allocation model is located after the level node associated with the resource prediction model; wherein, the resource prediction model is a resource prediction model trained based on the resource occupancy feature set, and the task allocation model is a task allocation model generated based on the task priority feature set.

[0014] A resource scheduling topology graph is a graphical structure used to describe the relationships between resource nodes and task scheduling paths. It organizes multiple resource nodes hierarchically according to certain rules, with each level representing a set of resource nodes with similar characteristics. Hierarchical nodes are interconnected through connection paths, which embed specific connection rules to guide resource allocation and task forwarding. A resource prediction model is a dynamic resource prediction model trained on a resource occupancy feature set. This feature set contains relevant characteristics of resource node resource occupancy over different time periods, such as fluctuations in processor load and memory usage. By learning these features, the resource prediction model can predict the future resource occupancy of a target resource node. For example, using a Long Short-Term Memory (LSTM) network architecture as a resource prediction model can effectively model time-series data and capture dynamic trends in resource occupancy. A task allocation model is an allocation decision model generated based on a task priority feature set. This feature set contains various priority-related parameters for tasks, such as task deadlines and the number of dependent tasks. The task allocation model assigns appropriate resources and execution order to tasks based on these parameters. For example, a decision tree model can be used as the task allocation model, which can make decisions based on different task priority parameters and allocate tasks to appropriate resource nodes.

[0015] In this embodiment, after acquiring the resource scheduling topology map, the device waits to receive task execution requests. When a task execution request is received, it carries task requirement parameters, which will be used in subsequent task parsing and resource allocation processes. For example, an execution request for a data processing task may carry task requirement parameters such as data size and processing time requirements.

[0016] As one implementation, before obtaining the resource scheduling topology map, a step of generating the resource scheduling topology map is also included, specifically including:

[0017] Step S101: Collect real-time running status data and historical task execution trajectories of multiple resource nodes; real-time running status data includes processor load rate, memory usage rate and network throughput; historical task execution trajectories include resource node identifiers of assigned tasks, task execution time windows and inter-task dependencies.

[0018] Real-time operational status data reflects the current operating status of resource nodes. Processor load rate indicates the workload of the processor at a given moment, memory utilization rate indicates memory usage, and network throughput indicates the network data transmission rate. Historical task execution trajectories record relevant information about tasks executed by resource nodes in the past. The resource node identifier of the assigned task is used to determine which resource node the task was executed on. The task execution time window represents the time range from the start to the end of the task, and the inter-task dependencies describe the sequence and dependencies between different tasks. For example, in a distributed computing device, monitoring tools collect real-time operational status data such as processor load rate, memory utilization rate, and network throughput of each computing node, while simultaneously retrieving historical task execution trajectories from the task management device, including the computing node identifier assigned to each task, the specific execution time of the task, and the dependencies between tasks.

[0019] Step S102: Extract the dynamic load feature set of each resource node from the real-time running status data; the dynamic load feature set includes the fluctuation period of processor load rate, the peak interval of memory occupancy rate, and the transmission stability index of network throughput.

[0020] Dynamic load feature sets are sets of features extracted from real-time operational status data that reflect the dynamic changes in the load of resource nodes. The fluctuation period of processor load rate represents the periodic changes in processor load rate over a certain period; the peak interval of memory utilization rate represents the time interval between peak memory utilization rates; and the transmission stability index of network throughput is used to measure the stability of network data transmission. For example, by analyzing processor load rate data over a period of time, its fluctuation period can be calculated; by statistically analyzing memory utilization rate data, its peak interval can be determined; and by evaluating network throughput data, a transmission stability index can be calculated. Time series analysis methods, such as Fourier transform, can be used to extract the fluctuation period of processor load rate.

[0021] Step S103: Extract the topological connection features of inter-task dependencies from the historical task execution trajectory; the topological connection features include the resource node call order, cross-node data transmission frequency, and synchronization parameters of task execution intervals in the same task chain.

[0022] Topological connectivity features are a set of features extracted from historical task execution trajectories to describe the dependencies between tasks. The resource node invocation order within the same task chain represents the order in which resource nodes are invoked; the cross-node data transmission frequency indicates the frequency of data transmission between different resource nodes; and the synchronization parameter of task execution intervals measures the consistency between different task execution intervals. For example, by analyzing task dependencies in historical task execution trajectories, the invocation order of resource nodes within the same task chain can be determined; the number of data transmissions between different resource nodes can be counted to calculate the cross-node data transmission frequency; and the synchronization parameter of task execution intervals can be evaluated. Graph theory methods can be used to analyze dependencies between tasks, construct a task dependency graph, and extract topological connectivity features from it.

[0023] Step S104: Fuse the dynamic load feature set with the topology connection features to generate a comprehensive topology feature vector for each resource node; the comprehensive topology feature vector is used to characterize the global attributes of the resource node in load dynamics and task collaboration relationships.

[0024] The comprehensive topology feature vector is a vector obtained by fusing dynamic load feature sets and topology connectivity features. It integrates information on the dynamic load changes and task collaboration relationships of resource nodes, providing a more comprehensive description of the global attributes of resource nodes. The fusion method can employ vector concatenation, sequentially linking the vectors of the dynamic load feature set and the topology connectivity features to form a new vector. For example, feature vectors from the dynamic load feature set, such as processor load rate fluctuation periods, peak memory usage intervals, and network throughput transmission stability indicators, can be concatenated with feature vectors from the topology connectivity feature set, such as resource node call order, cross-node data transmission frequency, and synchronization parameters of task execution intervals within the same task chain, to generate a comprehensive topology feature vector for each resource node.

[0025] Step S105: Perform hierarchical clustering analysis on multiple resource nodes based on the comprehensive topological feature vector to generate an initial hierarchical structure; each level in the initial hierarchical structure contains a set of resource nodes with similar comprehensive topological feature vectors, and the levels are arranged according to the tightness of task collaboration.

[0026] Hierarchical clustering analysis is a method for hierarchically grouping data objects, classifying them into different hierarchical structures based on the similarity between objects. In this step, hierarchical clustering analysis is performed on multiple resource nodes based on comprehensive topological feature vectors, grouping resource nodes with similar comprehensive topological feature vectors into the same level. Task collaboration tightness refers to the degree of cooperation between different resource nodes during task execution. Hierarchies are arranged according to task collaboration tightness, ensuring that resource nodes with close task collaboration are located in adjacent levels. For example, using hierarchical clustering algorithms, such as agglomerative hierarchical clustering, the similarity between resource nodes is calculated based on the comprehensive topological feature vectors. Resource nodes with high similarity are gradually merged to form different levels, ultimately generating the initial hierarchical structure.

[0027] As one implementation method, step S105 involves performing hierarchical clustering analysis on multiple resource nodes based on the comprehensive topological feature vector to generate an initial hierarchical structure, which may specifically include:

[0028] Step S1051: Input the comprehensive topological feature vector of each resource node into the feature mapping layer of the hierarchical partitioning model to generate a dimensionality-reduced low-dimensional feature representation; the low-dimensional feature representation retains the key difference attributes of the comprehensive topological feature vector.

[0029] A hierarchical partitioning model is used to hierarchically partition resource nodes. The feature mapping layer is a component of this model, its role being to map the high-dimensional composite topological feature vector to a low-dimensional space, generating a dimensionality-reduced low-dimensional feature representation. This low-dimensional feature representation reduces the feature dimensionality while preserving the key differentiating attributes of the composite topological feature vector, which can distinguish different resource nodes. For example, using Principal Component Analysis (PCA) as the feature mapping layer in a hierarchical partitioning model, the dimensionality of each resource node's composite topological feature vector is reduced, extracting the principal components that represent the differences between resource nodes and generating a low-dimensional feature representation.

[0030] Step S1052: Calculate the feature similarity between any two resource nodes based on the low-dimensional feature representation, and construct a resource node relationship graph based on the feature similarity; the edge weights in the resource node relationship graph are positively correlated with the feature similarity.

[0031] Feature similarity is a metric that measures the degree of feature similarity between two resource nodes. It is determined by calculating the distance between low-dimensional feature representations (such as Euclidean distance, cosine similarity, etc.). A resource node relationship graph is a graph structure with resource nodes as vertices and relationships between nodes as edges. The weight of each edge represents the feature similarity between the nodes. The higher the feature similarity, the greater the edge weight. For example, the cosine similarity between the low-dimensional feature representations of any two resource nodes can be calculated and used as a measure of feature similarity. A resource node relationship graph can be constructed based on feature similarity, and a graph database can be used to store and manage the resource node relationship graph.

[0032] Step S1053: Perform community discovery processing on the resource node relationship graph to identify multiple resource node communities; the edge weight within each resource node community is greater than the preset community threshold, and the edge weight between communities is less than the community threshold.

[0033] Community detection is a method for identifying sets of tightly connected nodes (i.e., communities) in a graph structure. A preset community threshold is a pre-defined threshold used to distinguish edge weights within and between communities. By performing community detection on a resource node graph, the nodes in the graph are divided into multiple resource node communities. Nodes within each community have high connection strength (edge ​​weights greater than the preset community threshold), while the connection strength between communities is low (edge ​​weights less than the community threshold). For example, using the Louvain algorithm to perform community detection on a resource node graph, resource nodes are divided into different communities based on edge weights and the preset community threshold.

[0034] Step S1054: Based on the size of each resource node community and the uniformity of the distribution of internal feature similarity, divide the resource node communities into core level or edge level; the community size corresponding to the core level is greater than the preset level size threshold and the distribution uniformity is higher than the preset uniformity threshold.

[0035] The core and peripheral layers are defined based on the size of the resource node community and the uniformity of its internal feature similarity distribution. Community size refers to the number of resource nodes within the community, while the uniformity of internal feature similarity distribution indicates the distribution of feature similarity among nodes within the community. Preset layer size thresholds and preset uniformity thresholds are pre-defined thresholds used to divide the community into core and peripheral layers. When the community size is greater than the preset layer size threshold and the uniformity of its internal feature similarity distribution is higher than the preset uniformity threshold, the community is classified as a core layer; otherwise, it is classified as a peripheral layer. For example, the number of nodes in each resource node community is counted as the community size, the variance of the feature similarity among nodes within the community is calculated as a measure of distribution uniformity, and the resource node community is classified into a core or peripheral layer based on the preset layer size threshold and preset uniformity threshold.

[0036] Step S1055: The core layer is used as the main layer of the initial hierarchical structure and arranged from high to low according to the tightness of task collaboration; the edge layer is used as the auxiliary layer of the initial hierarchical structure and the subordinate relationship is bound according to the feature similarity between it and the core layer.

[0037] The backbone layer is the main component of the initial hierarchical structure. It consists of core layers, arranged from high to low in terms of task collaboration tightness, ensuring that core layers with close task collaboration are arranged adjacently. The auxiliary layer consists of edge layers, which are bound to the core layers based on their feature similarity. Each edge layer is associated with one core layer, enabling it to provide auxiliary support. For example, core layers are sorted according to their task collaboration tightness, and the sorted core layers become the backbone layer. The feature similarity between edge layers and core layers is calculated, and the edge layers are bound to the core layers with the highest feature similarity.

[0038] Step S1056: Generate an initial hierarchical structure with multi-level nesting based on the arrangement and binding relationship between the main hierarchy and the auxiliary hierarchy; in the multi-level nesting relationship, each auxiliary hierarchy is associated with only one main hierarchy, and the weight of the association path is dynamically adjusted by the tightness of task collaboration between the two.

[0039] Multi-level nesting refers to the hierarchical nesting relationship between the main hierarchy and auxiliary hierarchy in the initial hierarchical structure. Each auxiliary hierarchy is associated with only one main hierarchy, and the weight of the association path reflects the tightness of task collaboration between the two, and this weight is dynamically adjusted according to the task collaboration situation. For example, based on the arrangement and binding relationship of the main hierarchy and auxiliary hierarchy, a multi-level nested hierarchical structure can be constructed. A dynamic programming algorithm can be used to dynamically adjust the weight of the association path according to the tightness of task collaboration, ultimately generating an initial hierarchical structure with multi-level nesting.

[0040] Step S106: Determine the connection rules between levels based on the task collaboration tightness of each level in the initial hierarchical structure; the connection rules include the communication path priority and task forwarding strategy of cross-level resource nodes.

[0041] Inter-level connection rules guide communication and task forwarding between resource nodes at different levels. Communication path priorities across resource nodes determine the order of priority for different communication paths; when communication between resource nodes at different levels is required, the device prioritizes the higher-priority communication path. Task forwarding strategies specify the forwarding methods and conditions for tasks between different levels. For example, when a task cannot be processed promptly at a resource node at a certain level, it is forwarded to a resource node at another level according to the task forwarding strategy. For instance, based on the task collaboration density at each level in the initial hierarchical structure, the communication cost and task processing efficiency between different levels are calculated to determine the communication path priorities across resource nodes; based on the task type and resource requirements, task forwarding strategies are formulated, such as prioritizing forwarding high-priority tasks to levels with richer resources.

[0042] Step S107: Based on the initial hierarchical structure and the connection rules between levels, generate a resource scheduling topology graph; each level node in the resource scheduling topology graph is associated with a set of resource nodes, and the connection paths between level nodes are embedded with connection rules.

[0043] A resource scheduling topology graph is a graphical structure generated based on an initial hierarchical structure and inter-hierarchical connection rules. Each hierarchical node represents a set of resource nodes, and the connection paths between hierarchical nodes reflect the connection rules. By embedding the connection rules into the connection paths, the resource scheduling topology graph can guide resource allocation and task scheduling. For example, the number and hierarchical relationships of hierarchical nodes are determined based on the initial hierarchical structure, and each hierarchical node is associated with its corresponding set of resource nodes; according to the inter-hierarchical connection rules, connection paths are established between hierarchical nodes, and information such as communication path priority and task forwarding strategies from the connection rules are embedded into the connection paths, ultimately generating the resource scheduling topology graph.

[0044] Step S200: Based on the task requirement parameters carried in the task execution request, parse the resource requirement type and execution constraints corresponding to the task execution request.

[0045] Task requirement parameters are detailed information about the task contained in the task execution request, reflecting the task's resource needs and execution conditions. Resource requirement type refers to the type of resources required by the task, such as computing resources, storage resources, and network resources. Execution constraints are limitations and requirements on the task execution process, such as resource upper limit thresholds, task parallelism limits, and expected progress thresholds. During the parsing process, the device analyzes and processes the task requirement parameters to extract the resource requirement type and execution constraints. For example, for a video encoding task, the task requirement parameters might explicitly indicate the need for certain computing power and storage space. By parsing these parameters, the device determines that the task's resource requirement type is computing resources and storage resources, and the execution constraints might include encoding time limits and resource usage limits.

[0046] Step S300: Starting from the initial level node of the resource scheduling topology graph, traverse all level nodes in the resource scheduling topology graph and obtain the resource prediction model or task allocation model associated with the currently traversed level node.

[0047] The starting level node of the resource scheduling topology graph is the starting point for resource scheduling, representing the initial set of resource nodes. Traversal refers to visiting each level node in the resource scheduling topology graph in a certain order. During traversal, the device obtains the resource prediction model or task allocation model associated with the currently traversed level node. If the current level node is associated with a resource prediction model, then this model will be used to predict the future resource occupancy of the resource node; if it is associated with a task allocation model, then this model will be used to determine the task allocation scheme. For example, in a resource scheduling topology graph with a multi-level structure, starting from the top-level starting level node, each level node is traversed downwards in sequence. For each level node, the corresponding resource prediction model or task allocation model is obtained according to its associated model type.

[0048] Step S400: Based on the resource requirement type and execution constraints, generate resource allocation instructions corresponding to the current traversal level node through the resource prediction model or task allocation model associated with the current traversal level node.

[0049] When the current traversal hierarchical node is associated with the resource prediction model, the device extracts a matching dynamic occupancy feature set from the historical resource dataset of the resource scheduling topology based on the resource demand type. Then, it performs real-time input adaptation on the resource prediction model to generate the predicted resource occupancy rate of the target resource node. Combined with the resource upper limit threshold in the execution constraints, it determines the available resource capacity of the target resource node. Finally, it generates a first resource allocation instruction based on the available resource capacity; for example, this instruction includes a list of allocable resource types and a capacity upper limit. When the current traversal hierarchical node is associated with the task allocation model, the device extracts priority parameters matching the task priority feature set from the task execution request. Based on these parameters, it generates task allocation weights and combines them with the task parallelism limit in the execution constraints to generate a second resource allocation instruction; for example, this instruction includes a task execution order identifier and a resource allocation ratio.

[0050] For example, for a computational task requiring significant computing resources, when traversing the hierarchical nodes of the associated resource prediction model, the device extracts a set of dynamic occupancy features related to computing resources from the historical resource dataset, such as fluctuations in processor load rate. It then predicts the computing resource occupancy rate of the target resource node using the resource prediction model. Based on the upper limit threshold of computing resources in the execution constraints, it determines the available computing resource capacity and generates a first resource allocation instruction containing the allocatable computing resource types and capacity limits. When traversing the hierarchical nodes of the associated task allocation model, the device extracts priority parameters such as task deadline and the number of dependent tasks from the task execution request, generates task allocation weights, and combines these with task parallelism constraints to generate a second resource allocation instruction containing the task execution order and resource allocation ratio.

[0051] As one implementation method, step S400, based on the resource requirement type and execution constraints, generates a resource allocation instruction corresponding to the current traversal level node through the resource prediction model or task allocation model associated with the current traversal level node. Specifically, this may include:

[0052] If the current traversal level node is associated with a resource prediction model, then the following steps are performed:

[0053] Step S410: Extract dynamic occupancy feature sets that match the resource demand type from the historical resource dataset of the resource scheduling topology graph.

[0054] Historical resource datasets store the historical resource occupancy of resource nodes, containing various resource occupancy data for different time periods. Dynamic occupancy feature sets are a set of features extracted from the historical resource dataset that match resource demand types, reflecting the dynamic occupancy of resource nodes in terms of resource demand types. For example, for a task requiring computing resources, dynamic occupancy feature sets related to computing resources are extracted from the historical resource dataset, such as processor load fluctuation characteristics and computing task execution time distribution. Specifically, firstly, resource occupancy curves of multiple resource nodes recorded in the historical resource dataset within a preset time period are obtained. Each resource occupancy curve is segmented, and the peak fluctuation characteristics, continuous occupancy duration characteristics, and load balancing characteristics corresponding to each segment are extracted. Then, based on the resource category identifier in the resource demand type, a target feature subset associated with the resource category identifier is selected from these features. Finally, the target feature subset is normalized to generate the dynamic occupancy feature set.

[0055] As one implementation method, step S410 involves extracting a dynamic occupancy feature set matching the resource demand type from the historical resource dataset of the resource scheduling topology. Specifically, this may include:

[0056] Step S411: Obtain the resource occupancy curves of multiple resource nodes centrally recorded in the historical resource dataset within a preset time period;

[0057] Historical resource datasets record the resource usage of multiple resource nodes over different time periods. Resource usage curves represent the changes in resource utilization rates of resource nodes over a preset time period. For example, by querying the historical resource dataset, one can obtain curves showing the changes in processor load rates of multiple computing nodes over the past week.

[0058] Step S412: Perform feature segmentation processing on each resource usage curve, and extract the peak fluctuation features, continuous usage duration features, and load balancing features corresponding to each curve segment;

[0059] Feature segmentation divides the resource usage curve into multiple feature segments, each with similar characteristics. Peak fluctuation characteristics represent the fluctuation of peak values ​​in the curve, such as peak height and fluctuation frequency; duration of continuous use characteristics represent the length of time the resource is continuously used; and load balancing characteristics represent the degree of balance in resource usage. For example, a sliding window method can be used to segment the resource usage curve, calculating peak fluctuation characteristics, duration of continuous use characteristics, and load balancing characteristics for each feature segment.

[0060] Step S413: Based on the resource category identifier in the resource demand type, filter out the target feature subset associated with the resource category identifier from the peak fluctuation characteristics, continuous occupancy duration characteristics, and load balancing characteristics;

[0061] The resource category identifier in the resource requirement type clearly defines the type of resources required by the task. Based on this identifier, a subset of target features associated with the extracted features is selected. For example, if the resource requirement type is computing resources, features related to computing resources, such as peak fluctuation characteristics of processor load rate and continuous occupation duration characteristics of computing tasks, are selected from peak fluctuation characteristics, continuous occupation duration characteristics, and load balancing characteristics.

[0062] Step S414: Normalize the target feature subset to generate a dynamically occupied feature set; wherein, the normalization process includes mapping feature values ​​of different dimensions to a unified numerical range and eliminating linear correlation between features.

[0063] Normalization is performed to eliminate dimensional differences and linear correlations between different features, making feature values ​​comparable. Normalizing a subset of the target features generates a dynamically occupied feature set. For example, min-max normalization can be used to map the feature values ​​of the target feature subset to the [0, 1] interval, while principal component analysis can be used to eliminate linear correlations between features, thus generating a dynamically occupied feature set.

[0064] Step S420: Adapt the resource prediction model to the dynamic occupancy feature set in real time to generate the predicted resource occupancy rate of the target resource node.

[0065] Real-time input adaptation refers to converting a dynamic occupancy feature set into a format and feature representation suitable for input to a resource prediction model. The resource prediction model predicts the future resource occupancy rate of a target resource node based on the input dynamic occupancy feature set. For example, the dynamic occupancy feature set is input into the feature encoding layer of the resource prediction model to generate a high-dimensional feature vector. The time series analysis layer of the resource prediction model then performs sliding window segmentation on the high-dimensional feature vector to obtain feature segments corresponding to multiple local time windows. Each feature segment is processed by convolutional kernel extraction to generate convolutional feature maps for each time window. These convolutional feature maps are then input into the long short-term memory (LSTM) network layer of the resource prediction model. The LSM network layer models the temporal dependencies of the convolutional feature maps, outputting a predicted resource occupancy rate sequence for the target resource node at future time points. Finally, the predicted resource occupancy rate is determined based on the maximum value and fluctuation trend in the predicted resource occupancy rate sequence.

[0066] As one implementation method, step S420 involves real-time input adaptation of the resource prediction model based on the dynamic occupancy feature set to generate the predicted resource occupancy rate of the target resource node. Specifically, this may include:

[0067] Step S421: Input the dynamic occupancy feature set into the feature encoding layer of the resource prediction model to generate a high-dimensional feature vector;

[0068] The feature encoding layer is a component of resource prediction models. Its role is to transform the dynamic occupancy feature set into a high-dimensional feature vector. The feature encoding layer encodes the dynamic occupancy feature set, extracting deeper information from the features and generating a high-dimensional feature vector. For example, a deep neural network can be used as the feature encoding layer. The dynamic occupancy feature set is input into the neural network, and after processing by multiple layers of neurons, a high-dimensional feature vector is generated.

[0069] Step S422: Through the time series analysis layer of the resource prediction model, the high-dimensional feature vector is segmented by a sliding window to obtain feature segments corresponding to multiple local time windows;

[0070] The time series analysis layer is used to process time series data, and sliding window segmentation is a method to divide time series data into multiple local time windows. By using sliding window segmentation, a high-dimensional feature vector is divided into feature segments corresponding to multiple local time windows, each containing feature information over a period of time. For example, a fixed-size sliding window can be used to segment a high-dimensional feature vector, with the window sliding a certain step size each time, resulting in multiple feature segments corresponding to local time windows.

[0071] Step S423: Perform convolution kernel extraction processing on each feature segment to generate convolution feature maps for each time window, and input the convolution feature maps into the long short-term memory network layer of the resource prediction model;

[0072] Kernel extraction is an operation in convolutional neural networks (CNNs). It involves performing convolution operations between a kernel and a feature segment to extract local features and generate a convolutional feature map. Long Short-Term Memory (LSTM) layers are neural networks capable of processing sequential data, capturing long-term dependencies in time-series data. The convolutional feature maps are input into the LTM layer to model the time-series data. For example, a CNN can be used to extract convolutional features from each feature segment, generating a convolutional feature map, which is then input into the LTM layer to model temporal dependencies.

[0073] Step S424: Model the temporal dependency relationship of the convolutional feature map through the long short-term memory network layer, and output the resource occupancy prediction sequence of the target resource node at future time points;

[0074] Long Short-Term Memory (LSTM) network layers model temporal dependencies in convolutional feature maps, learning long-term dependencies in time-series data and outputting a predicted sequence of resource occupancy rates for target resource nodes at future points in time. For example, based on the input convolutional feature maps, the LSM network layer filters and transmits information through internal gating mechanisms (such as input gate, forget gate, and output gate), learns the changing trends of resource occupancy rates, and outputs a predicted sequence of resource occupancy rates for a future period.

[0075] Step S425: Determine the predicted resource utilization rate based on the maximum value and fluctuation trend in the resource utilization rate prediction sequence;

[0076] The resource utilization prediction sequence contains predicted resource utilization values ​​for target resource nodes at multiple future time points. The final predicted resource utilization rate is determined based on the maximum value and fluctuation trend in this sequence. For example, at least one candidate peak is extracted from the resource utilization prediction sequence, and the change in resource utilization rate at each candidate peak's corresponding prediction time point and adjacent time windows is recorded. Based on the change in resource utilization rate, the fluctuation pattern type of the candidate peak (e.g., continuous upward, oscillating, and sudden spike) is identified. According to the fluctuation pattern type, the stability level of the candidate peak is determined. If the fluctuation pattern type of the candidate peak is continuous upward and the stability level is higher than a preset fluctuation threshold, the candidate peak is dynamically adjusted to generate an adjusted candidate peak. If the fluctuation pattern type of the candidate peak is oscillating and the stability level is lower than the preset fluctuation threshold, the original value of the candidate peak is retained and marked as a high-risk fluctuation area. Finally, the final predicted resource utilization rate is generated based on the adjusted candidate peak, the high-risk fluctuation area, and the unmarked candidate peak. The final predicted resource utilization rate is the weighted average of the adjusted candidate peak and the unmarked candidate peak, and the candidate peak corresponding to the high-risk fluctuation area is assigned the lowest weight.

[0077] As one implementation method, step S425, determining the predicted resource utilization rate based on the maximum value and fluctuation trend in the resource utilization rate prediction sequence, may specifically include:

[0078] Step S4251: Extract at least one candidate peak from the resource occupancy prediction sequence, and record the prediction time point and the change in resource occupancy within the adjacent time window corresponding to each candidate peak.

[0079] Candidate peaks are local maxima in the resource utilization prediction sequence. By traversing the resource utilization prediction sequence, all local maxima are identified as candidate peaks. The prediction time point corresponding to each candidate peak is recorded to understand when the peak occurs; simultaneously, the change in resource utilization within adjacent time windows is recorded to analyze the peak fluctuation. For example, for a resource utilization prediction sequence containing 100 time points, by comparing the resource utilization values ​​at adjacent time points, all local maxima are identified as candidate peaks, and the time point corresponding to each candidate peak and the change in resource utilization within adjacent time windows are recorded.

[0080] Step S4252: Based on the change in resource utilization rate, identify the fluctuation pattern type of the candidate peak; the fluctuation pattern types include continuous rising type, oscillating fluctuation type and sudden spike type.

[0081] Fluctuation pattern types are categorized based on the fluctuations of candidate peak values. The fluctuation pattern type is identified by analyzing the changes in resource utilization within adjacent time windows. A continuous upward fluctuation pattern indicates that resource utilization continues to rise over a period of time; an oscillating fluctuation pattern indicates that resource utilization fluctuates within a certain range; and a sudden spike pattern indicates that resource utilization suddenly experiences a very high peak. For example, the fluctuation pattern type of a candidate peak value can be determined based on the sign and magnitude of the changes in resource utilization within adjacent time windows. If the changes in resource utilization within several consecutive adjacent time windows are positive and gradually increase, it is a continuous upward fluctuation pattern; if the changes in resource utilization alternate between positive and negative and fluctuate within a certain range, it is an oscillating fluctuation pattern; and if the changes in resource utilization within a certain adjacent time window suddenly increase and are significantly higher than in other time windows, it is a sudden spike pattern.

[0082] Step S4253: Determine the stability level of the candidate peak based on the fluctuation pattern type; the stability level is used to characterize the probability of the candidate peak continuing at future time points.

[0083] The stability level is determined based on the fluctuation pattern type, with different fluctuation pattern types corresponding to different stability levels. Candidate peaks exhibiting a continuous upward trend typically have a higher stability level because their upward trend is relatively stable; candidate peaks exhibiting oscillating fluctuations have a relatively lower stability level because their fluctuations are more complex; and candidate peaks exhibiting sudden spikes have the lowest stability level because they appear abruptly and are difficult to predict. For example, a stability level range can be defined for each fluctuation pattern type, and the stability level of the candidate peak can be determined based on its fluctuation pattern type.

[0084] Step S4254: If the fluctuation pattern of the candidate peak is continuously rising and the stability level is higher than the preset fluctuation threshold, then the candidate peak is dynamically adjusted to generate an adjusted candidate peak; the dynamic adjustment includes reducing the numerical amplitude of the candidate peak according to the slope of the change in resource occupancy rate within adjacent time windows.

[0085] The preset fluctuation threshold is a pre-defined threshold used to judge the stability of candidate peak values. When the fluctuation pattern of a candidate peak value is continuously rising and its stability level is higher than the preset fluctuation threshold, the candidate peak value is dynamically adjusted to avoid overestimating resource utilization. The dynamic adjustment reduces the numerical amplitude of the candidate peak value based on the slope of the change in resource utilization within adjacent time windows. For example, if the slope of the change in resource utilization within adjacent time windows is large, it indicates that the resource utilization is rising rapidly; in this case, the numerical amplitude of the candidate peak value is appropriately reduced to generate an adjusted candidate peak value.

[0086] Step S4255: If the fluctuation pattern type of the candidate peak is oscillating fluctuation and the stability level is lower than the preset fluctuation threshold, then retain the original value of the candidate peak and mark it as a high-risk fluctuation area.

[0087] When a candidate peak exhibits an oscillating fluctuation pattern and its stability level is below a preset fluctuation threshold, it indicates that the candidate peak's fluctuation is relatively unstable and carries significant risk. In this case, the original value of the candidate peak is retained and marked as a high-risk fluctuation area for special handling in subsequent resource allocation. For example, during resource allocation, the resource utilization corresponding to the high-risk fluctuation area is assessed and processed more cautiously.

[0088] Step S4256: Generate the final predicted resource utilization rate based on the adjusted candidate peaks, high-risk fluctuation areas, and unlabeled candidate peaks; wherein, the final predicted resource utilization rate is the weighted average of the adjusted candidate peaks and the unlabeled candidate peaks, and the candidate peaks corresponding to the high-risk fluctuation areas are assigned the lowest weight.

[0089] The final predicted resource utilization rate is calculated based on the adjusted candidate peaks, high-risk fluctuation areas, and unlabeled candidate peaks. To reduce the impact of high-risk fluctuation areas on the prediction results, the candidate peaks corresponding to these areas are assigned the lowest weight. For example, a weighted average method is used, where the weighted average of the adjusted candidate peaks and the unlabeled candidate peaks is calculated based on the weight of each candidate peak to obtain the final predicted resource utilization rate.

[0090] Step S430: Determine the available resource capacity of the target resource node based on the predicted resource utilization rate and the resource upper limit threshold in the execution constraints.

[0091] The predicted resource utilization rate indicates the resource usage of a target resource node over a future period, while the resource cap threshold in the execution constraints limits resource usage. By comparing the predicted resource utilization rate and the resource cap threshold, the available resource capacity of the target resource node is determined. For example, if the predicted resource utilization rate is 80% and the resource cap threshold is 90%, then the available resource capacity of the target resource node is 10%.

[0092] Step S440: Generate a first resource allocation instruction based on the available resource capacity. The first resource allocation instruction includes a list of allocable resource types and a capacity limit.

[0093] The first resource allocation instruction is generated based on the available resource capacity of the target resource node. It specifies the list of allocable resource types and the capacity limit for each type of resource. For example, for a target resource node with computing and storage resources, the first resource allocation instruction is generated based on the available resource capacity. This instruction may contain a list of allocable computing resources (such as the number of CPU cores) and storage resources (such as disk space) types, as well as the corresponding capacity limits.

[0094] If the current traversal level node is associated with a task allocation model, then the following steps are executed:

[0095] Step S450: Extract priority parameters that match the task priority feature set from the task execution request. The priority parameters include the task deadline, the number of dependent tasks, and the resource exclusivity identifier.

[0096] A task priority feature set is a set of features used to measure task priority. Priority parameters are parameters extracted from the task execution request that match the task priority feature set. The task deadline indicates the exact time the task must be completed, the number of dependent tasks indicates the number of other tasks the task depends on, and the resource exclusivity flag indicates whether the task needs to exclusively occupy certain resources. For example, for the execution request of a data processing task, priority parameters such as the task deadline, the number of dependent tasks, and the resource exclusivity flag are extracted.

[0097] Step S460: Generate task allocation weights based on priority parameters.

[0098] Task allocation weights are weight values ​​calculated based on priority parameters to measure task priority. For example, the task urgency identifier, resource dependency parameter, and task relevance parameter are parsed from the task execution request. An initial priority weight is determined based on the task urgency identifier, and then adjusted a first time based on the resource dependency parameter. The data sharing degree with other tasks is determined based on the task relevance parameter, and a second adjustment is made to the adjusted initial priority weight based on the data sharing degree. Finally, the weight values ​​after both adjustments are standardized and mapped to generate the task allocation weights. The standardization mapping scales the weight values ​​proportionally to a preset weight range, ensuring that the sum of the allocation weights for tasks at the same level is a fixed value.

[0099] As one implementation method, step S460, generating task allocation weights based on priority parameters, may specifically include:

[0100] Step S461: Parse the task urgency identifier, resource dependency parameter, and task relevance parameter from the task execution request.

[0101] The task urgency flag indicates the urgency level of the task, the resource dependency parameter indicates the degree to which the task depends on resources, and the task relevance parameter indicates the degree of relevance between the task and other tasks. These parameters are extracted by parsing the task execution request. For example, for the execution request of a software development task, the task urgency flag (e.g., urgent, moderate, not urgent), resource dependency parameters (e.g., the amount of computing resources, storage resources, etc. required), and task relevance parameters (e.g., the dependency relationship with other development tasks) are parsed.

[0102] Step S462: Determine the initial priority weight based on the task urgency identifier, and make the first adjustment to the initial priority weight based on the resource dependency parameter.

[0103] The initial priority weight is determined based on the task urgency indicator; the higher the urgency, the greater the initial priority weight. Then, the initial priority weight is adjusted based on the resource dependency parameter. If the task is highly dependent on resources, the initial priority weight is increased; conversely, it is decreased. For example, for an urgent task, a high initial priority weight is set; if the task is also highly dependent on resources, the weight is appropriately increased based on the initial priority weight.

[0104] Step S463: Determine the data sharing degree with other tasks based on the task correlation parameter, and make a second adjustment to the adjusted initial priority weight based on the data sharing degree.

[0105] Data sharing degree is determined based on the task relevance parameter, representing the extent to which a task shares data with other tasks. The adjusted initial priority weights are then adjusted based on the data sharing degree. If the data sharing degree is high, the priority weight is increased to ensure data interaction and collaboration between tasks; conversely, the priority weight is decreased. For example, for tasks that share a significant amount of data with other tasks, the weight value is appropriately increased based on the adjusted initial priority weights.

[0106] Step S464: Standardize the weight values ​​after the two adjustments to generate task allocation weights; wherein, the standardization mapping includes scaling the weight values ​​proportionally to a preset weight range so that the sum of the task allocation weights at the same level is a fixed value.

[0107] Standardization mapping aims to ensure the comparability and consistency of task allocation weights at the same level. It scales the weight values, after two adjustments, proportionally to a preset weight range, ensuring that the sum of task allocation weights at the same level is a fixed value. For example, a min-max standardization method can be used to map weight values ​​to the [0, 1] range, while guaranteeing that the sum of all task allocation weights at the same level is 1, thus generating task allocation weights.

[0108] Step S470: Based on the task allocation weight and the task parallelism limit in the execution constraints, generate a second resource allocation instruction. The second resource allocation instruction includes the task execution order identifier and the resource allocation ratio.

[0109] Task parallelism limits the number of tasks that can be executed concurrently. Based on task allocation weights and the parallelism limit, the execution order and resource allocation ratio of the tasks are determined, generating a second resource allocation instruction. For example, for multiple tasks, the tasks are sorted according to their allocation weights, and combined with the parallelism limit, it is determined which tasks can be executed simultaneously and the resource allocation ratio for each task, generating a second resource allocation instruction that includes task execution order identifiers and resource allocation ratios.

[0110] Step S500: According to the resource allocation instruction, dynamically allocate the computing task corresponding to the task execution request to the target resource node, and trigger the target resource node to execute the computing task.

[0111] Resource allocation instructions specify the types and quantities of resources required for a task, as well as the execution order of the tasks. Based on these instructions, the device dynamically allocates the computational tasks corresponding to the task execution requests to appropriate target resource nodes. Dynamic allocation means that the device flexibly adjusts its allocation based on the real-time status of resources and the needs of the task to ensure efficient resource utilization. A target resource node is the resource node selected to execute the computational task. Once task allocation is complete, the device triggers the target resource node to begin executing the computational task. For example, for a database query task, the device, based on the resource allocation instructions, allocates the task to a target resource node with sufficient storage resources and computing power, and then sends an execution instruction to that target resource node, triggering it to begin executing the database query task.

[0112] As one implementation method, the method provided in this embodiment of the invention may further include the following steps:

[0113] Step S600: During the execution of the computing task on the target resource node, collect the actual resource usage data and task execution progress data of the target resource node in real time.

[0114] Actual resource usage data reflects the actual resource utilization of the target resource node when executing computing tasks, such as processor load rate and memory usage rate. Task execution progress data indicates the execution progress of computing tasks, such as the percentage of completed tasks and the remaining task load. By collecting this data in real time, it is possible to understand the task execution status and resource usage in a timely manner. For example, monitoring tools can be used to collect actual resource usage data such as processor load rate and memory usage rate of the target resource node in real time, while simultaneously obtaining task execution progress data from task management devices.

[0115] Step S700: Generate model optimization instructions for the resource prediction model based on the deviation between the actual resource occupancy data and the predicted resource occupancy rate.

[0116] The deviation value is the difference between the actual resource utilization data and the predicted resource utilization rate, reflecting the prediction error of the resource prediction model. Based on the deviation value, model optimization instructions are generated for the resource prediction model to optimize and adjust it, thereby improving the model's prediction accuracy. For example, if the actual resource utilization rate is significantly higher than the predicted resource utilization rate, it indicates that the model underestimates resource demand, and corresponding model optimization instructions are generated to adjust the model's parameters and structure.

[0117] Step S800: Generate parameter update instructions for the task allocation model based on the task execution progress data and the expected progress threshold in the execution constraints.

[0118] The expected progress threshold is the anticipated value of the task execution progress specified in the execution constraints. By comparing the task execution progress data with the expected progress threshold, it is determined whether the task is being executed as planned. If there is a deviation between the task execution progress and the expected progress threshold, a parameter update instruction is generated for the task allocation model to update the parameters and optimize the task allocation scheme. For example, if the task execution progress is significantly behind the expected progress threshold, it indicates that the task allocation is unreasonable, a parameter update instruction is generated to adjust the parameters of the task allocation model, and the task is reallocated.

[0119] Step S900: Based on the model optimization instructions and parameter update instructions, perform incremental training on the resource prediction model and task allocation model associated in the resource scheduling topology graph to generate an updated resource scheduling topology graph.

[0120] Incremental training refers to further training and optimizing the model using new data based on the existing model. Based on model optimization instructions and parameter update instructions, incremental training is performed on the resource prediction model and task allocation model associated with the resource scheduling topology graph to improve model performance and adaptability. For example, actual resource occupancy data is merged with historical resource datasets to generate an expanded resource dataset. Noise filtering is applied to the expanded resource dataset, and a new dynamic occupancy feature set is extracted from it. This new dynamic occupancy feature set is then input into the online learning layer of the resource prediction model. The convolutional kernel parameters of the resource prediction model are updated through gradient backpropagation in the online learning layer, while the parameters of the long short-term memory network layer are frozen. Task execution progress data is compared with a preset progress deviation threshold. If the deviation exceeds the threshold, the node to be split is located in the decision tree layer of the task allocation model. Based on the task correlation parameters in the task execution progress data, the node to be split is re-partitioned using features, generating an updated task allocation model. Finally, an updated resource scheduling topology graph is generated based on the updated model.

[0121] Step S1000: When a new task execution request is received, resource allocation is performed using the updated resource scheduling topology.

[0122] When the device receives a new task execution request, it uses the updated resource scheduling topology to allocate resources, improving the accuracy and efficiency of resource allocation. The association model in the updated resource scheduling topology has been incrementally trained, enabling it to better adapt to the actual situation of resource nodes and the needs of the task, thus achieving more reasonable resource allocation. For example, for a new computing task execution request, based on the updated resource scheduling topology, by traversing hierarchical nodes, calling the associated resource prediction model and task allocation model, a resource allocation instruction is generated, and the task is assigned to a suitable target resource node.

[0123] As one implementation method, step S900 involves incremental training of the resource prediction model and task allocation model associated with the resource scheduling topology graph, which may specifically include:

[0124] Step S910: Merge the actual resource usage data with the historical resource dataset to generate an extended resource dataset, and perform noise filtering on the extended resource dataset.

[0125] Actual resource usage data is collected in real time during the execution of computational tasks on the target resource node. This data is then merged with the historical resource dataset to obtain the extended resource dataset. Noise filtering is performed to remove noise and outliers from the extended resource dataset, improving data quality. For example, statistical methods (such as mean filtering and median filtering) are used to filter noise from the extended resource dataset, removing abnormal fluctuations and noise points.

[0126] Step S920: Extract the newly added dynamic occupancy feature set from the extended resource dataset and input the newly added dynamic occupancy feature set into the online learning layer of the resource prediction model;

[0127] The newly added dynamic occupancy feature set is a set of features extracted from the extended resource dataset that matches the resource demand type, and it contains new resource occupancy information. This new dynamic occupancy feature set is input into the online learning layer of the resource prediction model, allowing the model to learn and adjust based on the new data. For example, using a method similar to step S410, the new dynamic occupancy feature set is extracted from the extended resource dataset and input into the online learning layer of the resource prediction model.

[0128] Step S930: Update the convolution kernel parameters of the resource prediction model through gradient backpropagation using the online learning layer, while freezing the parameters of the long short-term memory network layer;

[0129] Backpropagation is an optimization algorithm used to update model parameters. It calculates the gradient of the loss function with respect to the model parameters and updates the parameters to reduce the value of the loss function. During incremental training, gradient backpropagation updates the convolutional kernel parameters of the resource prediction model through online learning layers to adapt to new data. Simultaneously, to maintain model stability, the parameters of the Long Short-Term Memory (LSTM) network layers are frozen and not updated. For example, stochastic gradient descent (SGD) can be used to update the convolutional kernel parameters of the resource prediction model using gradient backpropagation.

[0130] Step S940: Compare the task execution progress data with the preset progress deviation threshold. If the deviation exceeds the threshold, locate the node to be split from the decision tree layer of the task allocation model.

[0131] The preset progress deviation threshold is a pre-defined threshold used to determine whether the task execution progress is normal. The task execution progress data is compared with this threshold; if the deviation exceeds the threshold, it indicates that the parameters of the task allocation model may need adjustment. The node to be split is located in the decision tree layer of the task allocation model; this node is the one that needs further partitioning. For example, when using a decision tree algorithm to build a task allocation model, if the task execution progress deviation exceeds the threshold, the node that needs to be split is identified from the nodes of the decision tree.

[0132] Step S950: Based on the task correlation parameters in the task execution progress data, perform feature re-division on the nodes to be split and generate an updated task allocation model.

[0133] The task correlation parameter represents the degree of association between a task and other tasks. Based on the task correlation parameter, the nodes to be split are re-partitioned according to their features, and the partitioning rules of the nodes are redefined to improve the accuracy of the task allocation model. For example, based on information such as data sharing and dependencies in the task correlation parameter, the nodes to be split are re-partitioned according to their features to generate an updated task allocation model.

[0134] As one implementation method, the method provided in this embodiment of the invention may further include the following steps:

[0135] Step S1100: When a conflict is detected between resource allocation instructions for multiple task execution requests, obtain the conflict resolution model from the conflict handling node of the resource scheduling topology graph;

[0136] Resource allocation instruction conflict refers to a contradiction or competition between resource allocation instructions requested by multiple tasks, such as multiple tasks requesting the same resource. Conflict handling nodes are nodes in the resource scheduling topology specifically designed to handle conflicts, and conflict resolution models are models used to resolve resource allocation instruction conflicts. When a conflict is detected, the conflict resolution model is retrieved from the conflict handling node. For example, during resource scheduling, resource allocation instructions from multiple task execution requests are monitored in real time; when a conflict is detected, a pre-trained conflict resolution model is retrieved from the conflict handling node in the resource scheduling topology.

[0137] Step S1200: Evaluate the impact of conflicts by using a conflict resolution model to assess the priority parameters, resource requirement types, and historical execution records of multiple task execution requests;

[0138] Conflict impact assessment refers to evaluating the degree to which a conflict affects task execution and resource allocation. The conflict resolution model assesses the impact of conflicts based on the priority parameters, resource requirement types, and historical execution records of multiple task execution requests. For example, it extracts task type tags, resource occupancy duration, and dependent resource lists from each task execution request. Based on the task type tags, it determines the business domain to which the task belongs and obtains the historical conflict resolution strategies for that business domain. Based on these strategies and resource occupancy durations, it calculates the resource release cost for each task execution request. Based on the overlap of dependent resource lists, it determines the resource competition intensity between tasks. Finally, it combines the resource release cost and competition intensity to generate a conflict level score and recommended solutions in the conflict impact assessment results.

[0139] As one implementation method, step S1200 involves evaluating the conflict impact of multiple task execution requests based on priority parameters, resource requirement types, and historical execution records using a conflict resolution model. Specifically, this may include:

[0140] Step S1210: Extract the task type tag, resource usage duration, and dependent resource list from each task execution request;

[0141] Task type tags identify the type of task, such as computation tasks, storage tasks, etc.; resource usage duration indicates the length of time the task consumes resources; the dependent resource list lists the resources the task depends on. This information is extracted from each task execution request to provide data support for subsequent conflict impact assessments. For example, for a data processing task execution request, the task type tag (e.g., data processing task), resource usage duration (e.g., estimated execution time), and dependent resource list (e.g., required computational resources, storage resources, etc.) are extracted.

[0142] Step S1220: Determine the business domain to which the task belongs based on the task type label, and obtain the historical conflict resolution strategies of the business domain;

[0143] A business domain refers to the business area to which a task belongs. Different business domains may have different conflict resolution strategies. The business domain to which a task belongs is determined based on its task type label, and then historical conflict resolution strategies for that business domain are retrieved from a database or knowledge base. For example, if the task type label is "financial trading task," then the business domain to which the task belongs is determined to be the financial domain, and historical conflict resolution strategies are retrieved from the financial domain's knowledge base.

[0144] Step S1230: Based on historical conflict resolution strategies and resource occupation duration, calculate the resource release cost for each task execution request;

[0145] Resource release cost refers to the cost incurred in releasing resources occupied by a task, including costs related to task interruption and data loss. Based on historical conflict resolution strategies and resource occupation duration, the resource release cost for each task execution request is calculated. For example, based on the resource release cost assessment methods for different types of tasks in historical conflict resolution strategies, combined with the task's resource occupation duration, the resource release cost for each task execution request is calculated.

[0146] Step S1240: Determine the intensity of resource contention between tasks based on the degree of overlap in the dependent resource lists;

[0147] The degree of overlap in resource dependency lists indicates the extent to which multiple tasks require the same resource; the higher the overlap, the stronger the resource contention. The intensity of resource contention between tasks is determined by comparing their resource dependency lists. For example, the intersection of the resource dependency lists of multiple tasks can be calculated, and the size of the intersection determines the intensity of resource contention.

[0148] Step S1250: Based on the combined resource release cost and resource competition intensity, generate the conflict level score and recommended solution in the conflict impact assessment results.

[0149] Conflict severity rating is a quantitative assessment of the severity of a conflict, while recommended solutions are suggestions for resolving the conflict based on the conflict severity rating and the specific circumstances of the task. The conflict severity rating and recommended solutions are generated by combining resource release costs and resource competition intensity in the conflict impact assessment results. For example, a weighted average method can be used to calculate the conflict severity rating by combining resource release costs and resource competition intensity. Based on the conflict severity rating and information such as task priority, recommended solutions, such as resource preemption or task queuing, are then provided.

[0150] Step S1300: Based on the conflict impact assessment results, generate a resource preemption instruction or a task queuing instruction; wherein, the resource preemption instruction is used to forcibly release the resources occupied by low-priority tasks, and the task queuing instruction is used to add conflicting tasks to the waiting queue and allocate resources in order.

[0151] Based on the conflict impact assessment results, the severity of the conflict and the priority of the tasks are determined, and corresponding instructions are generated. If the conflict is severe and low-priority tasks are occupying critical resources, a resource preemption instruction is generated to forcibly release the resources occupied by low-priority tasks to meet the needs of high-priority tasks. If the conflict is not severe, the conflicting tasks can be added to a waiting queue, and resources can be allocated in order. In this case, a task queuing instruction is generated. For example, for cases with a high conflict level score, a resource preemption instruction is generated to forcibly release the resources occupied by low-priority tasks; for cases with a low conflict level score, a task queuing instruction is generated to add the conflicting tasks to the waiting queue.

[0152] Step S1400: Feed back the resource preemption instruction or task queuing instruction to the target resource node and update the task status identifier in the resource scheduling topology diagram.

[0153] The generated resource preemption or task queuing instructions are fed back to the target resource node, allowing it to execute the corresponding operations. Simultaneously, the task status flags in the resource scheduling topology are updated to reflect the latest task status. For example, sending a resource preemption instruction to the target resource node allows it to release resources occupied by low-priority tasks; sending a task queuing instruction to the target resource node adds conflicting tasks to the waiting queue. The task status flags in the resource scheduling topology are also updated, such as changing the status flag of low-priority tasks to "resource preempted" and the status flag of conflicting tasks to "queuing".

[0154] As one implementation method, the method provided in this embodiment of the invention may further include the following steps:

[0155] Step S1500: When initially constructing the resource scheduling topology map, collect hardware configuration data, network topology, and historical task execution logs of multiple resource nodes.

[0156] Hardware configuration data reflects the hardware performance of resource nodes, such as processor model, memory capacity, and disk capacity; network topology describes the network connections between resource nodes; and historical task execution logs record detailed information about tasks executed by resource nodes in the past. This data is collected during the initial construction of the resource scheduling topology map to provide foundational data for subsequent resource node clustering analysis and model training. For example, hardware configuration data can be obtained by querying the hardware management devices of resource nodes, network topology discovery tools can be used to obtain the network topology, and historical task execution logs can be retrieved from task management devices.

[0157] Step S1600: Extract the computing power characteristics, storage capacity characteristics, and network bandwidth characteristics of resource nodes from the hardware configuration data.

[0158] Computational capability features represent the computing power of resource nodes, such as the number of processor cores and clock speed; storage capacity features represent the storage capacity of resource nodes, such as disk capacity and memory capacity; network bandwidth features represent the network transmission capability of resource nodes, such as network bandwidth and transmission rate. These features are extracted from hardware configuration data for subsequent cluster analysis. For example, the number of processor cores and clock speed are extracted as computational capability features, disk capacity and memory capacity are extracted as storage capacity features, and network bandwidth and transmission rate are extracted as network bandwidth features.

[0159] Step S1700: Extract task execution duration features, resource utilization features, and failed task distribution features from historical task execution logs.

[0160] The task execution duration feature represents the length of time a task takes to execute; the resource utilization feature represents the resource efficiency of resource nodes during task execution; and the failed task distribution feature represents the distribution of failed tasks across different resource nodes and times. These features are extracted from historical task execution logs for subsequent cluster analysis. For example, the execution time of each task is calculated as the task execution duration feature; processor utilization and memory utilization of resource nodes during task execution are calculated as resource utilization features; and the distribution of failed tasks across different resource nodes and times is analyzed as the failed task distribution feature.

[0161] Step S1800: Based on the characteristics of computing power, storage capacity, network bandwidth, task execution time, resource utilization, and distribution of failed tasks, perform cluster analysis on resource nodes to generate initial hierarchical node partitioning results.

[0162] Cluster analysis is a method for grouping data objects. Based on characteristics such as computing power, storage capacity, network bandwidth, task execution time, resource utilization, and failed task distribution, it performs cluster analysis on resource nodes, grouping resource nodes with similar characteristics into the same group to generate an initial hierarchical node partitioning result. For example, using the K-means clustering algorithm, the similarity between resource nodes is calculated based on these characteristics, and resource nodes with high similarity are grouped together to obtain the initial hierarchical node partitioning result.

[0163] Step S1900: Based on the initial hierarchical node partitioning results, assign an initial resource prediction model or task allocation model to each hierarchical node, and generate a resource scheduling topology map by training through historical data playback.

[0164] Based on the initial hierarchical node partitioning, an initial resource prediction model or task allocation model is assigned to each hierarchical node. The resource prediction model is used to predict the future resource occupancy of resource nodes, while the task allocation model is used to determine the task allocation scheme. These models are trained and optimized using historical data replay to generate a resource scheduling topology. For example, an initial resource prediction model and task allocation model are assigned to each hierarchical node, and historical data such as historical task execution logs and hardware configuration data are used to train the models, adjust the model parameters, and finally generate a resource scheduling topology.

[0165] In summary, the AI-based multi-line collaborative scheduling method provided by this invention achieves dynamic resource allocation and scheduling of tasks by constructing a resource scheduling topology graph and combining a resource prediction model and a task allocation model. Simultaneously, by incrementally training the model using real-time data acquisition and handling resource allocation instruction conflicts, the accuracy and efficiency of resource allocation are improved, enabling it to better adapt to the needs of different tasks and the actual conditions of resource nodes.

[0166] Please see Figure 2 , Figure 2 This is a schematic diagram of a multi-line collaborative scheduling device provided in an embodiment of the present invention. The multi-line collaborative scheduling device includes at least a processor 101, a communication interface 102, and a memory 103. The processor 101, communication interface 102, and memory 103 can be connected via a bus or other means. The processor 101 (or Central Processing Unit, CPU) is the computing and control core of the multi-line collaborative scheduling device, capable of parsing various instructions and processing various data within the device. The communication interface 102 may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.), and can be used for sending and receiving data under the control of the processor 101; the communication interface 102 can also be used for data transmission and interaction within the multi-line collaborative scheduling device. The memory 103 is the storage device in the multi-line collaborative scheduling device, used to store programs and data. It is understood that the memory 103 here can include the built-in memory of the multi-line collaborative scheduling device, or it can include extended memory supported by the device. The memory 103 provides storage space for storing the operating system of the multi-line collaborative scheduling device, which may include, but is not limited to, Android, iOS, Windows Phone, etc. The present invention does not limit this.

[0167] In one embodiment, the processor 101 executes the AI-based multi-line cooperative scheduling method provided above in the embodiments of the present invention by running a computer program in the memory 103.

Claims

1. A multi-line cooperative scheduling method based on artificial intelligence, characterized in that, The method includes: Collect real-time running status data and historical task execution trajectories from multiple resource nodes; the real-time running status data includes processor load rate, memory usage rate, and network throughput; the historical task execution trajectory includes resource node identifiers of assigned tasks, task execution time windows, and inter-task dependencies; Extract the dynamic load feature set of each resource node from the real-time operating status data; the dynamic load feature set includes the fluctuation period of processor load rate, the peak interval of memory occupancy rate, and the transmission stability index of network throughput. Extract the topological connection features of inter-task dependencies from the historical task execution trajectory; the topological connection features include the resource node call order, cross-node data transmission frequency, and synchronization parameters of task execution intervals in the same task chain; The dynamic load feature set is fused with the topology connection features to generate a comprehensive topology feature vector for each resource node; the comprehensive topology feature vector is used to characterize the global attributes of the resource node in load dynamics and task collaboration relationships. Based on the comprehensive topological feature vector, hierarchical clustering analysis is performed on multiple resource nodes to generate an initial hierarchical structure; each level in the initial hierarchical structure contains a set of resource nodes with similar comprehensive topological feature vectors, and the levels are arranged according to the tightness of task collaboration. Based on the task collaboration tightness of each level in the initial hierarchical structure, the connection rules between levels are determined; the connection rules include the communication path priority and task forwarding strategy of cross-level resource nodes; Based on the initial hierarchical structure and the connection rules between the hierarchical levels, a resource scheduling topology graph is generated; each hierarchical node in the resource scheduling topology graph is associated with a set of resource nodes, and the connection paths between hierarchical nodes are embedded in the connection rules. The resource scheduling topology is obtained and a task execution request is received. Each level node of the resource scheduling topology is associated with a resource prediction model or a task allocation model. In each execution path of the resource scheduling topology, the level node associated with the task allocation model is located after the level node associated with the resource prediction model. The resource prediction model is a resource prediction model trained based on a resource occupancy feature set, and the task allocation model is a task allocation model generated based on a task priority feature set. Based on the task requirement parameters carried in the task execution request, parse the resource requirement type and execution constraints corresponding to the task execution request; Starting from the initial level node of the resource scheduling topology graph, traverse all level nodes in the resource scheduling topology graph to obtain the resource prediction model or task allocation model associated with the currently traversed level node. Based on the resource demand type and execution constraints, a resource allocation instruction corresponding to the current traversal level node is generated through the resource prediction model or task allocation model associated with the current traversal level node. According to the resource allocation instruction, the computing task corresponding to the task execution request is dynamically allocated to the target resource node, and the target resource node is triggered to execute the computing task.

2. The method according to claim 1, characterized in that, The step of generating a resource allocation instruction corresponding to the current traversal level node based on the resource demand type and execution constraints, through the resource prediction model or task allocation model associated with the current traversal level node, includes: If the currently traversed hierarchical node is associated with a resource prediction model, then the following steps are performed: Extract a dynamic occupancy feature set that matches the resource demand type from the historical resource dataset of the resource scheduling topology; The resource prediction model is adapted in real time based on the dynamic occupancy feature set to generate the predicted resource occupancy rate of the target resource node; Based on the predicted resource utilization rate and the resource upper limit threshold in the execution constraints, the available resource capacity of the target resource node is determined; A first resource allocation instruction is generated based on the available resource capacity, the first resource allocation instruction including a list of allocable resource types and a capacity limit; If the currently traversed hierarchical node is associated with a task allocation model, then the following steps are performed: Extract priority parameters that match the task priority feature set from the task execution request. The priority parameters include the task deadline, the number of dependent tasks, and the resource exclusivity identifier. Task allocation weights are generated based on the priority parameters; Based on the task allocation weights and the task parallelism limits in the execution constraints, a second resource allocation instruction is generated. The second resource allocation instruction includes a task execution order identifier and a resource allocation ratio.

3. The method according to claim 2, characterized in that, The step of extracting a dynamic occupancy feature set matching the resource demand type from the historical resource dataset of the resource scheduling topology includes: Obtain the resource occupancy curves of multiple resource nodes recorded in the historical resource dataset within a preset time period; Each resource usage curve is segmented to extract the peak fluctuation characteristics, continuous usage duration characteristics, and load balancing characteristics of each segment. Based on the resource category identifier in the resource demand type, a subset of target features associated with the resource category identifier is selected from the peak fluctuation characteristics, continuous occupancy duration characteristics, and load balancing characteristics; The target feature subset is normalized to generate the dynamic occupancy feature set; wherein, the normalization process includes mapping feature values ​​of different dimensions to a unified numerical range and eliminating linear correlation between features.

4. The method according to claim 2, characterized in that, The step of performing real-time input adaptation of the resource prediction model based on the dynamic occupancy feature set to generate the predicted resource occupancy rate of the target resource node includes: The dynamic occupancy feature set is input into the feature encoding layer of the resource prediction model to generate a high-dimensional feature vector. The high-dimensional feature vector is segmented by a sliding window through the time series analysis layer of the resource prediction model to obtain feature segments corresponding to multiple local time windows. Each feature segment is processed by convolution kernel extraction to generate convolution feature maps for each time window, and the convolution feature maps are input into the long short-term memory network layer of the resource prediction model. The convolutional feature map is modeled using the long short-term memory network layer to perform temporal dependency modeling, and the predicted resource occupancy sequence of the target resource node at future time points is output. The predicted resource utilization rate is determined based on the maximum value and fluctuation trend in the predicted resource utilization rate sequence.

5. The method according to claim 2, characterized in that, The step of generating task allocation weights based on the priority parameters includes: The task urgency identifier, resource dependency parameter, and task relevance parameter are parsed from the task execution request. The initial priority weight is determined based on the task urgency identifier, and the initial priority weight is adjusted for the first time based on the resource dependency parameter; The data sharing degree with other tasks is determined based on the task relevance parameter, and the adjusted initial priority weight is adjusted a second time based on the data sharing degree. The weight values, after two adjustments, are standardized and mapped to generate the task allocation weights; wherein, the standardization mapping includes scaling the weight values ​​proportionally to a preset weight range so that the sum of the task allocation weights at the same level is a fixed value.

6. The method according to claim 1, characterized in that, The method further includes: During the execution of the computing task on the target resource node, the actual resource usage data and task execution progress data of the target resource node are collected in real time. Based on the deviation between the actual resource occupancy data and the predicted resource occupancy rate, model optimization instructions for the resource prediction model are generated. Based on the task execution progress data and the expected progress threshold in the execution constraints, a parameter update instruction for the task allocation model is generated. Based on the model optimization instructions and parameter update instructions, the resource prediction model and task allocation model associated in the resource scheduling topology graph are incrementally trained to generate an updated resource scheduling topology graph. When a new task execution request is received, resource allocation is performed using the updated resource scheduling topology. The incremental training of the resource prediction model and task allocation model associated with the resource scheduling topology graph includes: The actual resource usage data is merged with the historical resource dataset to generate an extended resource dataset, and noise filtering is applied to the extended resource dataset. The newly added dynamic occupancy feature set is extracted from the extended resource dataset, and the newly added dynamic occupancy feature set is input into the online learning layer of the resource prediction model; The parameters of the convolutional kernels of the resource prediction model are updated by gradient backpropagation through the online learning layer, while the parameters of the long short-term memory network layer are frozen. The task execution progress data is compared with a preset progress deviation threshold. If the deviation exceeds the threshold, the node to be split is located in the decision tree layer of the task allocation model. Based on the task correlation parameters in the task execution progress data, the nodes to be split are re-partitioned to generate an updated task allocation model.

7. The method according to claim 1, characterized in that, The method further includes: When a conflict is detected between resource allocation instructions for multiple task execution requests, a conflict resolution model is obtained from the conflict handling node of the resource scheduling topology. The conflict resolution model is used to assess the conflict impact of the priority parameters, resource requirement types, and historical execution records of the multiple task execution requests. Based on the conflict impact assessment results, a resource preemption instruction or a task queuing instruction is generated; wherein, the resource preemption instruction is used to forcibly release the resources occupied by low-priority tasks, and the task queuing instruction is used to add conflicting tasks to the waiting queue and allocate resources in order. The resource preemption instruction or task queuing instruction is fed back to the target resource node, and the task status identifier in the resource scheduling topology is updated.

8. The method according to claim 7, characterized in that, The assessment of the conflict impact of the multiple task execution requests based on priority parameters, resource requirement types, and historical execution records using the conflict resolution model includes: Extract the task type tag, resource usage duration, and dependent resource list from each task execution request; The business domain to which the task belongs is determined based on the task type label, and the historical conflict resolution strategies of the business domain are obtained. Based on the historical conflict resolution strategies and resource occupation duration, calculate the resource release cost for each task execution request; The intensity of resource contention between tasks is determined based on the degree of overlap in the dependent resource lists. By combining the resource release costs and resource competition intensity, a conflict level score and recommended solution are generated in the conflict impact assessment results.

9. The method according to claim 1, characterized in that, The method further includes: During the initial construction of the resource scheduling topology, hardware configuration data, network topology, and historical task execution logs of multiple resource nodes are collected. Extract the computing power characteristics, storage capacity characteristics, and network bandwidth characteristics of the resource nodes from the hardware configuration data; Extract task execution duration characteristics, resource utilization characteristics, and failed task distribution characteristics from the historical task execution logs; Based on the aforementioned computing power characteristics, storage capacity characteristics, network bandwidth characteristics, task execution time characteristics, resource utilization characteristics, and failed task distribution characteristics, cluster analysis is performed on resource nodes to generate initial hierarchical node partitioning results. Based on the initial hierarchical node division results, an initial resource prediction model or task allocation model is assigned to each hierarchical node, and the resource scheduling topology is generated by training through historical data replay.

10. A multi-line collaborative scheduling system, characterized in that, include: A memory, wherein a computer program is stored; A processor for loading the computer program to implement the AI-based multi-line cooperative scheduling method as described in any one of claims 1-9.

Images