Multi-line cooperative scheduling method and device based on artificial intelligence

Abstract

The invention provides a multi-line collaborative scheduling method and device based on artificial intelligence, and the method comprises the steps: obtaining a resource scheduling topological graph, receiving a task execution request, and analyzing a resource demand type and an execution constraint condition corresponding to the task execution request according to a task demand parameter carried in the task execution request; traversing all hierarchical nodes in the resource scheduling topological graph from an initial hierarchical node of the resource scheduling topological graph, and obtaining a resource prediction model or a task allocation model associated with the current traversed hierarchical node; based on the resource demand type and the execution constraint condition, generating a resource allocation instruction corresponding to the current traversal hierarchy node through a resource prediction model or a task allocation model associated with the current traversal hierarchy node; and according to the resource allocation instruction, dynamically allocating a calculation task corresponding to the task execution request to the target resource node, and triggering the target resource node to execute the calculation task. According to the invention, the resource allocation precision and the system robustness can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular, to a multi-line collaborative scheduling method and device based on artificial intelligence. Background Art

[0002] In the AI middle platform, multi-line collaboration is required to optimize the execution efficiency of computing tasks and resource utilization through dynamic resource allocation. In the existing scheduling methods, a static resource pool partitioning strategy or a task allocation based on a single decision model (such as only relying on resource load metrics or task priorities) is usually adopted, and resource prediction and task allocation are treated as independent links during the scheduling process. However, in the existing methods, the resource prediction model cannot capture the fluctuating characteristics of dynamic resource occupancy in real time (such as the load peak caused by burst tasks), resulting in the prediction result lagging behind the actual resource state. The task allocation model may allocate high-priority tasks to nodes that are about to be fully loaded due to the lack of pre-resource capacity evaluation, leading to task execution interruption or resource contention conflicts. At the same time, the resource prediction and task allocation links in the traditional method are separated and cannot form a closed-loop feedback, so that the scheduling strategy cannot be dynamically optimized according to real-time task execution data, resulting in an increase in resource idle rate and an exacerbation of the risk of task deadline default. Summary of the Invention

[0003] The present invention provides a multi-line collaborative scheduling method and device based on artificial intelligence.

[0004] In a first aspect, an embodiment of the present invention provides a multi-line collaborative scheduling method based on artificial intelligence, the method including: obtaining a resource scheduling topology graph and receiving a task execution request; each level node of the resource scheduling topology graph is associated with a resource prediction model or a task allocation model, and in each execution path of the resource scheduling topology graph, the level node associated with the task allocation model is located after the level node associated with the resource prediction model; wherein, the resource prediction model is a resource prediction model trained based on a resource occupancy feature set, and the task allocation model is a task allocation model generated based on a task priority feature set; parsing the resource demand type and execution constraint conditions corresponding to the task execution request according to the task demand parameters carried in the task execution request; starting from the starting level node of the resource scheduling topology graph, traversing all level nodes in the resource scheduling topology graph, and obtaining the resource prediction model or the task allocation model associated with the currently traversed level node; generating a resource allocation instruction corresponding to the currently traversed level node through the resource prediction model or the task allocation model associated with the currently traversed level node based on the resource demand type and the execution constraint conditions; dynamically allocating the computing task corresponding to the task execution request to a target resource node according to the resource allocation instruction, and triggering the target resource node to execute the computing task.

[0005] In a second aspect, an embodiment of the present invention provides a multi-line collaborative scheduling device, including: a memory in which a computer program is stored; a processor configured to load the computer program to implement the multi-line collaborative scheduling method based on artificial intelligence as described above.

[0006] The multi-line collaborative scheduling method based on artificial intelligence provided by the present invention obtains a resource scheduling topology graph and analyzes the resource requirement types and execution constraint conditions in the task execution request, traverses all hierarchical nodes associated with resource prediction models or task allocation models starting from the starting hierarchical node of the topology graph, and generates resource allocation instructions based on a dynamic resource prediction and priority-driven allocation strategy; wherein, the resource prediction model predicts the available resource capacity of the target resource node by analyzing the dynamic occupancy feature set of the historical resource data set, and the task allocation model generates task allocation weights by extracting task deadline, dependency relationship, and resource exclusivity parameters in the task priority feature set, and dynamically allocates computing tasks to the target resource node in combination with the execution constraint conditions. In this way, the resource prediction model can accurately predict the dynamic load trend of the resource node based on the time series characteristics, avoiding allocation failures caused by resource fluctuations. The task allocation model adjusts the task scheduling weight through multi-dimensional priority parameters to ensure that high-urgency tasks obtain resource supply first; by sequentially connecting resource prediction and task allocation in the hierarchical nodes of the topology graph to form a "prediction - decision" closed loop, the resource capacity evaluation and task priority matching are realized in real-time collaboration, so as to achieve double optimization of resource utilization efficiency and task execution timeliness in complex multi-task scenarios; at the same time, incremental training is performed on the resource prediction model and the task allocation model based on the real-time collected task execution data, continuously optimizing the self-adaptability of the scheduling strategy of the topology graph, and further improving the resource allocation accuracy and system robustness. In addition, through the conflict resolution model, the conflict impact degree of resource competition tasks is evaluated and dynamically arbitrated, effectively reducing task queuing delay and resource preemption cost, and ensuring the efficient collaborative execution of multi-line tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0008] Figure 1 It is a flowchart of a multi-line collaborative scheduling method based on artificial intelligence provided by an embodiment of the present invention.

[0009] Figure 2 It is a schematic diagram of the composition of a multi-line collaborative scheduling device provided by an embodiment of the present invention. Detailed implementation manners

[0010] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0011] Please refer to Figure 1 , Figure 1 which is a flowchart of a multi-line collaborative scheduling method based on artificial intelligence provided by an embodiment of the present invention. The multi-line collaborative scheduling method based on artificial intelligence can be executed by a multi-line collaborative scheduling device, and the multi-line collaborative scheduling method based on artificial intelligence may include the following steps: The present invention provides a multi-line collaborative scheduling method based on artificial intelligence, and the method includes the following steps: Step S100: Obtain a resource scheduling topology graph and receive a task execution request; each hierarchical node of the resource scheduling topology graph is associated with a resource prediction model or a task assignment model, and in each execution path of the resource scheduling topology graph, the hierarchical node associated with the task assignment model is located after the hierarchical node associated with the resource prediction model; wherein, the resource prediction model is a resource prediction model trained based on a resource occupancy feature set, and the task assignment model is a task assignment model generated based on a task priority feature set.

[0012] The resource scheduling topology graph is a graphical structure used to describe the relationships between resource nodes and the task scheduling paths. It hierarchically organizes multiple resource nodes according to certain rules, and each level node represents a set of resource nodes with similar characteristics. The level nodes are interconnected by connection paths, and these connection paths embed specific connection rules for guiding resource allocation and task forwarding. The resource prediction model is a dynamic resource prediction model trained based on the resource occupancy feature set, which includes relevant features of the resource occupancy of resource nodes at different time periods, such as the fluctuation features of the processor load rate, memory occupancy rate, etc. By learning these features, the resource prediction model can predict the future resource occupancy rate of the target resource node. For example, the long short-term memory network (LSTM) architecture is used as the resource prediction model, which can effectively model time series data and capture the dynamic change trends of resource occupancy. The task allocation model is an allocation decision model generated based on the task priority feature set, which includes various priority-related parameters of tasks, such as task deadlines, the number of dependent tasks, etc. The task allocation model allocates appropriate resources and execution sequences for tasks according to these parameters. For example, a decision tree model can be used as the task allocation model, which can make decisions based on different task priority parameters and allocate tasks to appropriate resource nodes.

[0013] In this embodiment, after obtaining the resource scheduling topology graph, the device waits to receive a task execution request. When a task execution request is received, the request carries task requirement parameters, which will be used in the subsequent task parsing and resource allocation processes. For example, the execution request of a data processing task may carry task requirement parameters such as the data volume size and processing time requirements.

[0014] As an implementation, before obtaining the resource scheduling topology graph, it further includes the step of generating the resource scheduling topology graph, specifically including: Step S101: Collect the real-time operating status data and historical task execution trajectories of multiple resource nodes; the real-time operating status data includes the processor load rate, memory occupancy rate, and network throughput; the historical task execution trajectories include the resource node identifiers of the allocated tasks, the task execution time window, and the dependencies between tasks.

[0015] The real-time running status data reflects the running condition of the resource node at the current moment. The processor load rate represents the workload degree of the processor at a certain moment. The memory occupancy rate represents the usage of the memory. The network throughput represents the rate of network data transmission. The historical task execution track records the relevant information of the tasks executed by the resource node in the past. The resource node identifier of the allocated task is used to determine on which resource node the task is executed. The task execution time window represents the time range of the start and end of the task. The inter-task dependency relationship describes the sequence and dependency relationship between different tasks. For example, in a distributed computing device, the real-time running status data such as the processor load rate, memory occupancy rate, and network throughput of each computing node is collected through monitoring tools, and at the same time, the historical task execution track is obtained from the task management device, including the computing node identifier allocated to each task, the specific time of task execution, and the dependency relationship between tasks.

[0016] Step S102: Extract the dynamic load feature set of each resource node from the real-time running status data; the dynamic load feature set includes the fluctuation period of the processor load rate, the peak interval of the memory occupancy rate, and the transmission stability index of the network throughput.

[0017] The dynamic load feature set is a feature set extracted from the real-time running status data that can reflect the dynamic changes of the resource node load. The fluctuation period of the processor load rate represents the periodic change of the processor load rate within a certain period of time. The peak interval of the memory occupancy rate represents the time interval when the memory occupancy rate reaches the peak. The transmission stability index of the network throughput is used to measure the stability of network data transmission. For example, by analyzing the processor load rate data over a period of time, its fluctuation period is calculated; the memory occupancy rate data is statistically analyzed to determine its peak interval; the network throughput data is evaluated to calculate the transmission stability index. Time series analysis methods such as Fourier transform can be used to extract the fluctuation period of the processor load rate.

[0018] Step S103: Extract the topological connection features of the inter-task dependency relationship from the historical task execution track; the topological connection features include the resource node call order in the same task chain, the cross-node data transmission frequency, and the synchronization parameter of the task execution interval.

[0019] The topological connection features are a set of features extracted from historical task execution traces to describe the dependency relationships between tasks. The resource node call order in the same task chain represents the sequence in which each resource node is called in a task chain; the cross-node data transfer frequency represents the frequency of data transfer between different resource nodes; the synchronization parameter of task execution intervals is used to measure the consistency of different task execution intervals. For example, by analyzing the task dependency relationships in historical task execution traces, the call order of resource nodes in the same task chain is determined; the number of data transfers between different resource nodes is counted to calculate the cross-node data transfer frequency; the task execution intervals are analyzed to evaluate their synchronization parameters. Graph theory methods can be used to analyze the task dependency relationships, construct a task dependency graph, and extract topological connection features from it.

[0020] Step S104: Fuse the dynamic load feature set with the topological connection features to generate a comprehensive topological feature vector for each resource node; the comprehensive topological feature vector is used to characterize the global attributes of the resource node in terms of load dynamics and task collaboration relationships.

[0021] The comprehensive topological feature vector is a vector obtained by fusing the dynamic load feature set and the topological connection features. It synthesizes the information on the dynamic changes in the load of the resource node and the task collaboration relationships, and can more comprehensively describe the global attributes of the resource node. The fusion method can adopt the way of vector splicing, connecting the vectors of the dynamic load feature set and the topological connection features in sequence to form a new vector. For example, the feature vectors such as the processor load rate fluctuation period, the memory occupancy peak interval, and the network throughput transmission stability index in the dynamic load feature set are spliced with the feature vectors such as the resource node call order in the same task chain, the cross-node data transfer frequency, and the synchronization parameter of task execution intervals in the topological connection feature set to generate the comprehensive topological feature vector for each resource node.

[0022] Step S105: Perform hierarchical clustering analysis on multiple resource nodes based on the comprehensive topological feature vector to generate an initial hierarchical structure; each level in the initial hierarchical structure contains a set of resource nodes with similar comprehensive topological feature vectors, and the levels are arranged according to the task collaboration tightness.

[0023] Hierarchical clustering analysis is a method of hierarchically grouping data objects, which divides them into different hierarchical structures according to the similarity between objects. In this step, hierarchical clustering analysis is performed on multiple resource nodes based on the comprehensive topological feature vector, and resource nodes with similar comprehensive topological feature vectors are divided into the same level. The task collaboration tightness refers to the degree of collaboration between different resource nodes during task execution. The levels are arranged according to the task collaboration tightness, so that resource nodes with close task collaboration are located in adjacent levels. For example, a hierarchical clustering algorithm, such as the agglomerative hierarchical clustering algorithm, is used to calculate the similarity between resource nodes according to the comprehensive topological feature vector, and resource nodes with higher similarity are gradually merged to form different levels, and finally an initial hierarchical structure is generated.

[0024] As an implementation manner, in step S105, hierarchical clustering analysis is performed on multiple resource nodes based on the comprehensive topological feature vector to generate an initial hierarchical structure, which may specifically include: Step S1051: Input the comprehensive topological feature vector of each resource node into the feature mapping layer of the hierarchical division model to generate a low-dimensional feature representation after dimensionality reduction; the low-dimensional feature representation retains the key differential attributes of the comprehensive topological feature vector.

[0025] The hierarchical division model is a model used for hierarchical division of resource nodes. The feature mapping layer is a component of the hierarchical division model. Its function is to map the high-dimensional comprehensive topological feature vector into a low-dimensional space to generate a low-dimensional feature representation after dimensionality reduction. While reducing the feature dimension, the low-dimensional feature representation retains the key differential attributes of the comprehensive topological feature vector, and these key differential attributes can distinguish different resource nodes. For example, the principal component analysis (PCA) method is used as the feature mapping layer of the hierarchical division model to perform dimensionality reduction processing on the comprehensive topological feature vector of each resource node, extract the principal components that can represent the differences of resource nodes, and generate a low-dimensional feature representation.

[0026] Step S1052: According to the low-dimensional feature representation, calculate the feature similarity between any two resource nodes, and construct a resource node relationship graph based on the feature similarity; the edge weight in the resource node relationship graph is positively correlated with the feature similarity.

[0027] Feature similarity is an indicator to measure the degree of feature similarity between two resource nodes, which is determined by calculating the distance between low-dimensional feature representations (such as Euclidean distance, cosine similarity, etc.). The resource node relationship graph is a graphical structure that takes resource nodes as vertices, the relationships between nodes as edges, and the weight of an edge represents the feature similarity between nodes. The higher the feature similarity, the greater the weight of the edge. For example, calculate the cosine similarity between the low-dimensional feature representations of any two resource nodes, use it as a measure of feature similarity, construct a resource node relationship graph based on the feature similarity, and use a graph database to store and manage the resource node relationship graph.

[0028] Step S1053: Perform community discovery processing on the resource node relationship graph to identify multiple resource node communities; the edge weights within each resource node community are greater than a preset community threshold, and the edge weights between communities are less than the community threshold.

[0029] Community discovery processing is a method to identify a set of nodes with tight connections (i.e., communities) in a graph structure. The preset community threshold is a threshold set in advance to distinguish the edge weights within and between communities. By performing community discovery processing on the resource node relationship graph, the nodes in the graph are divided into multiple resource node communities. The nodes within each community have a high connection strength (edge weights are greater than the preset community threshold), while the connection strength between communities is low (edge weights are less than the community threshold). For example, use the Louvain algorithm to perform community discovery processing on the resource node relationship graph, and divide the resource nodes into different communities according to the edge weights and the preset community threshold.

[0030] Step S1054: Divide the resource node communities into a core level or a marginal level according to the scale of each resource node community and the distribution uniformity of the internal feature similarity; the community corresponding to the core level has a scale greater than the preset level scale threshold and a distribution uniformity higher than the preset uniformity threshold. The core level and the marginal level are divided according to the scale of the resource node community and the distribution uniformity of the internal feature similarity. The community scale refers to the number of resource nodes within the community, and the distribution uniformity of the internal feature similarity represents the distribution of the feature similarity between nodes within the community. The preset level scale threshold and the preset uniformity threshold are thresholds set in advance for dividing the core level and the marginal level. When the community scale is greater than the preset level scale threshold and the distribution uniformity of the internal feature similarity is higher than the preset uniformity threshold, the community is divided into the core level; otherwise, it is divided into the marginal level. For example, count the number of nodes in each resource node community as the community scale, calculate the variance of the feature similarity between nodes within the community as a measure of the distribution uniformity, and divide the resource node communities into the core level or the marginal level according to the preset level scale threshold and the preset uniformity threshold.

[0031] Step S1055: Take the core level as the backbone level of the initial level structure and arrange it in descending order according to the task collaboration tightness; take the edge level as the auxiliary level of the initial level structure and bind the affiliation relationship according to the feature similarity with the core level; The backbone level is the main component of the initial level structure. It is composed of the core level and is arranged in descending order according to the task collaboration tightness, so that the core levels with close task collaboration are arranged adjacent to each other. The auxiliary level is composed of the edge level, and the affiliation relationship is bound according to the feature similarity with the core level, that is, each edge level establishes an affiliation relationship with a core level, so that the edge level can provide auxiliary support for the core level. For example, sort the core levels according to the task collaboration tightness between them, and take the sorted core levels as the backbone level; calculate the feature similarity between the edge level and the core level, and bind the affiliation relationship between the edge level and the core level with the highest feature similarity.

[0032] Step S1056: Generate an initial level structure with a multi-level nested relationship according to the arrangement and binding relationship between the backbone level and the auxiliary level; in the multi-level nested relationship, each auxiliary level is only associated with one backbone level, and the weight of the association path is dynamically adjusted by the task collaboration tightness between the two.

[0033] The multi-level nested relationship refers to the hierarchical nested relationship between the backbone level and the auxiliary level in the initial level structure. Each auxiliary level is only associated with one backbone level, and the weight of the association path reflects the task collaboration tightness between the two, and this weight will be dynamically adjusted according to the task collaboration situation. For example, according to the arrangement order and binding relationship between the backbone level and the auxiliary level, construct a multi-level nested hierarchical structure, and use the dynamic programming algorithm to dynamically adjust the weight of the association path according to the task collaboration tightness, and finally generate an initial level structure with a multi-level nested relationship.

[0034] Step S106: Determine the connection rules between levels according to the task collaboration tightness of each level in the initial level structure; the connection rules include the communication path priority and task forwarding strategy of cross-level resource nodes.

[0035] The connection rules between levels are used to guide the communication and task forwarding of resource nodes between different levels. The communication path priority of cross-level resource nodes determines the priority order of different communication paths. When communication is required between resource nodes at different levels, the device preferentially selects the communication path with a higher priority. The task forwarding strategy stipulates the forwarding methods and conditions of tasks between different levels. For example, when a task cannot be processed in a timely manner at a resource node at a certain level, it is forwarded to resource nodes at other levels according to the task forwarding strategy. For example, according to the task cooperation tightness of each level in the initial level structure, calculate the communication cost and task processing efficiency between different levels, and determine the communication path priority of cross-level resource nodes; according to the type and resource requirements of the task, formulate a task forwarding strategy, such as preferentially forwarding high-priority tasks to levels with richer resources.

[0036] Step S107: Generate a resource scheduling topology graph based on the initial level structure and the connection rules between levels; in the resource scheduling topology graph, each level node is associated with a set of resource nodes, and the connection paths between level nodes embed the connection rules.

[0037] The resource scheduling topology graph is a graphical structure generated based on the initial level structure and the connection rules between levels. Each level node represents a set of resource nodes, and the connection paths between level nodes reflect the connection rules between levels. By embedding the connection rules into the connection paths, the resource scheduling topology graph can guide the allocation of resources and the scheduling of tasks. For example, determine the number and hierarchical relationship of level nodes according to the initial level structure, associate each level node with the corresponding set of resource nodes; according to the connection rules between levels, establish connection paths between level nodes, and embed information such as the communication path priority and task forwarding strategy in the connection rules into the connection paths, and finally generate the resource scheduling topology graph.

[0038] Step S200: Parse the resource requirement type and execution constraint conditions corresponding to the task execution request according to the task requirement parameters carried in the task execution request.

[0039] The task requirement parameters are the detailed information about the task included in the task execution request, which reflects the resource requirements of the task and the execution conditions. The resource requirement type refers to the types of resources required by the task, such as computing resources, storage resources, network resources, etc. The execution constraint conditions are some restrictions and requirements on the task execution process, such as resource upper limit thresholds, task parallelism limits, expected progress thresholds, etc. During the parsing process, the device analyzes and processes the task requirement parameters to extract the resource requirement type and the execution constraint conditions. For example, for a video encoding task, the task requirement parameters may clearly indicate the need for a certain amount of computing power and storage space. The device determines that the resource requirement type of this task is computing resources and storage resources through parsing these parameters, and the execution constraint conditions may include encoding time limits, resource usage upper limits, etc.

[0040] Step S300: Starting from the starting-level node of the resource scheduling topology graph, traverse all the level nodes in the resource scheduling topology graph to obtain the resource prediction model or task allocation model associated with the currently traversed level node.

[0041] The starting-level node of the resource scheduling topology graph is the starting point of resource scheduling, representing the initial set of resource nodes. Traversal means accessing each level node in the resource scheduling topology graph in a certain order. During the traversal process, the device obtains the resource prediction model or task allocation model associated with the currently traversed level node. If the currently traversed level node is associated with a resource prediction model, then this model will be used to predict the future resource occupancy of the resource nodes; if it is associated with a task allocation model, then this model will be used to determine the task allocation scheme. For example, in a resource scheduling topology graph with a multi-level structure, starting from the topmost starting-level node, traverse each level node downwards in turn. For each level node, obtain the corresponding resource prediction model or task allocation model according to the type of the associated model.

[0042] Step S400: Based on the resource requirement type and the execution constraint conditions, generate a resource allocation instruction corresponding to the currently traversed level node through the resource prediction model or task allocation model associated with the currently traversed level node.

[0043] When the currently traversed hierarchical node is associated with a resource prediction model, the device extracts a matching dynamic occupancy feature set from the historical resource dataset of the resource scheduling topology graph according to the resource demand type, then performs real-time input adaptation on the resource prediction model to generate the predicted resource occupancy rate of the target resource node, and then combines the resource upper limit threshold in the execution constraint conditions to determine the available resource capacity of the target resource node. Finally, a first resource allocation instruction is generated according to the available resource capacity. For example, this instruction includes a list of allocable resource types and a capacity upper limit. When the currently traversed hierarchical node is associated with a task allocation model, the device extracts priority parameters matching the task priority feature set from the task execution request, generates task allocation weights according to these parameters, and then combines the task parallelism limit in the execution constraint conditions to generate a second resource allocation instruction. For example, this instruction includes a task execution order identifier and a resource allocation ratio.

[0044] For example, for a computing task that requires a large amount of computing resources, when traversing to the hierarchical node associated with the resource prediction model, the device extracts a dynamic occupancy feature set related to computing resources from the historical resource dataset, such as the fluctuation feature of the processor load rate, etc. The device predicts the computing resource occupancy rate of the target resource node through the resource prediction model, determines the available computing resource capacity according to the computing resource upper limit threshold in the execution constraint conditions, and generates a first resource allocation instruction including the allocable computing resource type and the capacity upper limit. When traversing to the hierarchical node associated with the task allocation model, the device extracts priority parameters such as the task deadline and the number of dependent tasks from the task execution request, generates task allocation weights, and combines the task parallelism limit to generate a second resource allocation instruction including the task execution order and the resource allocation ratio.

[0045] As an implementation manner, step S400, based on the resource demand type and the execution constraint conditions, generates a resource allocation instruction corresponding to the currently traversed hierarchical node through the resource prediction model or the task allocation model associated with the currently traversed hierarchical node, and specifically may include: If the currently traversed hierarchical node is associated with a resource prediction model, then the following steps are executed: Step S410: Extract a dynamic occupancy feature set matching the resource demand type from the historical resource dataset of the resource scheduling topology graph.

[0046] The historical resource dataset is a dataset that stores the historical resource occupancy of resource nodes, and it contains various resource occupancy data of resource nodes in different time periods. The dynamic occupancy feature set is a set of features extracted from the historical resource dataset that matches the resource demand type, and it can reflect the dynamic occupancy of resource nodes in terms of resource demand type. For example, for a task that requires computing resources, a dynamic occupancy feature set related to computing resources is extracted from the historical resource dataset, such as the fluctuation feature of the processor load rate, the distribution of computing task execution time, etc. Specifically, first, obtain the resource occupancy curves of multiple resource nodes recorded in the historical resource dataset within a preset time period, perform feature segmentation processing on each resource occupancy curve, extract the peak fluctuation feature, continuous occupancy duration feature, and load balancing feature corresponding to each segment of the curve, then, according to the resource category identifier in the resource demand type, screen out the target feature subset associated with the resource category identifier from these features, and finally, perform normalization processing on the target feature subset to generate the dynamic occupancy feature set.

[0047] As an implementation, step S410, extracting a dynamic occupancy feature set that matches the resource demand type from the historical resource dataset of the resource scheduling topology map, may specifically include: Step S411: Obtain the resource occupancy curves of multiple resource nodes recorded in the historical resource dataset within a preset time period; The historical resource dataset records the resource occupancy of multiple resource nodes in different time periods. The resource occupancy curve is a curve that shows how the resource occupancy rate of a resource node changes over time within a preset time period. For example, by querying the historical resource dataset, obtain the curve of the processor load rate of multiple computing nodes changing over time in the past week.

[0048] Step S412: Perform feature segmentation processing on each resource occupancy curve, and extract the peak fluctuation feature, continuous occupancy duration feature, and load balancing feature corresponding to each segment of the curve; Feature segmentation processing is to divide the resource occupancy curve into multiple feature segments, and each feature segment has similar features. The peak fluctuation feature represents the fluctuation of the peak value in the curve, such as the height of the peak value, the fluctuation frequency, etc.; the continuous occupancy duration feature represents the length of time that the resource is continuously occupied; the load balancing feature represents the degree of balance of resource occupancy. For example, use the sliding window method to perform feature segmentation processing on the resource occupancy curve, and calculate the peak fluctuation feature, continuous occupancy duration feature, and load balancing feature for each feature segment.

[0049] Step S413: According to the resource category identifier in the resource demand type, screen out the target feature subset associated with the resource category identifier from the peak fluctuation feature, continuous occupancy duration feature, and load balancing feature; The resource category identifier in the resource requirement type clarifies the type of resources required for the task, and based on this identifier, the associated target feature subset is filtered out from the extracted features. For example, if the resource requirement type is computing resources, the features related to computing resources are filtered out from the peak fluctuation feature, the duration of continuous occupancy feature, and the load balancing feature, such as the peak fluctuation feature of the processor load rate, the duration of continuous occupancy feature of computing tasks, etc.

[0050] Step S414: Normalize the target feature subset to generate a dynamic occupancy feature set; where the normalization process includes mapping feature values with different dimensions to a unified numerical interval and eliminating the linear correlation between features.

[0051] The normalization process is to eliminate the dimensional difference and linear correlation between different features, making the feature values comparable. After normalizing the target feature subset, a dynamic occupancy feature set is generated. For example, the minimum-maximum normalization method is used to map the feature values of the target feature subset to the [0, 1] interval, and at the same time, the principal component analysis method is used to eliminate the linear correlation between features to generate a dynamic occupancy feature set.

[0052] Step S420: Perform real-time input adaptation on the resource prediction model according to the dynamic occupancy feature set to generate the predicted resource occupancy rate of the target resource node.

[0053] Real-time input adaptation means converting the dynamic occupancy feature set into a format and feature representation suitable for the input of the resource prediction model. The resource prediction model predicts the future resource occupancy rate of the target resource node based on the input dynamic occupancy feature set. For example, the dynamic occupancy feature set is input into the feature encoding layer of the resource prediction model to generate a high-dimensional feature vector. Through the time series analysis layer of the resource prediction model, the high-dimensional feature vector is segmented by a sliding window to obtain feature segments corresponding to multiple local time windows. Each feature segment is processed by a convolutional kernel extraction to generate a convolutional feature map for each time window, and the convolutional feature map is input into the long short-term memory network layer of the resource prediction model. The long short-term memory network layer models the temporal dependence relationship of the convolutional feature map and outputs a prediction sequence of the resource occupancy rate of the target resource node at future time points. Finally, based on the maximum value and the fluctuation trend in the resource occupancy rate prediction sequence, the predicted resource occupancy rate is determined.

[0054] As an implementation, step S420, performing real-time input adaptation on the resource prediction model according to the dynamic occupancy feature set to generate the predicted resource occupancy rate of the target resource node, may specifically include: Step S421: Input the dynamic occupancy feature set into the feature encoding layer of the resource prediction model to generate a high-dimensional feature vector; The feature encoding layer is a component of the resource prediction model, and its function is to convert the dynamic occupancy feature set into a high-dimensional feature vector. By encoding the dynamic occupancy feature set through the feature encoding layer, the deep information of the features is extracted to generate a high-dimensional feature vector. For example, a deep neural network is used as the feature encoding layer, and the dynamic occupancy feature set is input into the neural network. After being processed by multiple layers of neurons, a high-dimensional feature vector is generated.

[0055] Step S422: Through the time series analysis layer of the resource prediction model, perform sliding window segmentation on the high-dimensional feature vector to obtain feature segments corresponding to multiple local time windows; The time series analysis layer is used to process time series data. Sliding window segmentation is a method of dividing time series data into multiple local time windows. Through sliding window segmentation, the high-dimensional feature vector is divided into feature segments corresponding to multiple local time windows, and each feature segment contains the feature information within a certain period of time. For example, a fixed-size sliding window is used to segment the high-dimensional feature vector, and the window slides a certain step length each time to obtain feature segments corresponding to multiple local time windows.

[0056] Step S423: Perform convolution kernel extraction processing on each feature segment to generate a convolution feature map for each time window, and input the convolution feature map into the long short-term memory network layer of the resource prediction model; Convolution kernel extraction processing is an operation in a convolutional neural network. By performing a convolution operation between the convolution kernel and the feature segment, the local features of the feature segment are extracted to generate a convolution feature map. The long short-term memory network layer is a neural network that can process sequence data, and it can capture the long-term dependencies in time series data. Inputting the convolution feature map into the long short-term memory network layer allows it to model the time series data. For example, a convolutional neural network is used to perform convolution kernel extraction processing on each feature segment to generate a convolution feature map, and the convolution feature map is input into the long short-term memory network layer for modeling the temporal dependency relationship.

[0057] Step S424: Through the long short-term memory network layer, perform temporal dependency relationship modeling on the convolution feature map, and output the resource occupancy rate prediction sequence of the target resource node at future time points; The long short-term memory network layer performs temporal dependency relationship modeling on the convolution feature map, learns the long-term dependencies in the time series data, and outputs the resource occupancy rate prediction sequence of the target resource node at future time points. For example, based on the input convolution feature map, the long short-term memory network layer screens and transmits information through internal gating mechanisms (such as input gate, forget gate, output gate), learns the change trend of the resource occupancy rate, and outputs the resource occupancy rate prediction sequence for a future period of time.

[0058] Step S425: Determine the predicted resource occupancy rate according to the maximum value and the fluctuation trend in the resource occupancy rate prediction sequence. The resource occupancy rate prediction sequence contains the predicted resource occupancy rate values of the target resource node at multiple future time points. The final predicted resource occupancy rate is determined according to the maximum value and the fluctuation trend in this sequence. For example, extract at least one candidate peak from the resource occupancy rate prediction sequence, and record the predicted time point corresponding to each candidate peak and the change in the resource occupancy rate within the adjacent time window. Based on the change in the resource occupancy rate, identify the type of fluctuation pattern of the candidate peak (such as continuously rising type, oscillating fluctuation type, and sudden spike type). According to the type of fluctuation pattern, determine the stability level of the candidate peak. If the fluctuation pattern type of the candidate peak is continuously rising type and the stability level is higher than the preset fluctuation threshold, then dynamically adjust the candidate peak to generate an adjusted candidate peak. If the fluctuation pattern type of the candidate peak is oscillating fluctuation type and the stability level is lower than the preset fluctuation threshold, then retain the original value of the candidate peak and mark it as a high-risk fluctuation area. Finally, generate the final predicted resource occupancy rate according to the adjusted candidate peak, the high-risk fluctuation area, and the unmarked candidate peaks. The final predicted resource occupancy rate is the weighted average of the adjusted candidate peak and the unmarked candidate peaks, and the candidate peak corresponding to the high-risk fluctuation area is given the lowest weight.

[0059] As an implementation manner, in step S425, to determine the predicted resource occupancy rate according to the maximum value and the fluctuation trend in the resource occupancy rate prediction sequence, it may specifically include: Step S4251: Extract at least one candidate peak from the resource occupancy rate prediction sequence, and record the predicted time point corresponding to each candidate peak and the change in the resource occupancy rate within the adjacent time window.

[0060] The candidate peak is the local maximum value in the resource occupancy rate prediction sequence. By traversing the resource occupancy rate prediction sequence, find all the local maximum values as candidate peaks. Record the predicted time point corresponding to each candidate peak to know when the peak appears; at the same time, record the change in the resource occupancy rate within the adjacent time window for analyzing the fluctuation of the peak. For example, for a resource occupancy rate prediction sequence containing 100 time points, find all the local maximum values as candidate peaks by comparing the resource occupancy rate values at adjacent time points, and record the time point corresponding to each candidate peak and the change in the resource occupancy rate within the adjacent time window.

[0061] Step S4252: Identify the type of fluctuation pattern of the candidate peak based on the change in the resource occupancy rate; the type of fluctuation pattern includes continuously rising type, oscillating fluctuation type, and sudden spike type.

[0062] The fluctuation mode type is classified according to the fluctuations of candidate peaks, and the change amount of resource occupancy rate within adjacent time windows is analyzed to identify the fluctuation mode type. The continuously rising type means that the resource occupancy rate continuously rises within a period of time; the oscillating fluctuation type means that the resource occupancy rate fluctuates within a certain range; the sudden spike type means that a very high peak suddenly appears in the resource occupancy rate. For example, according to the positive or negative and magnitude of the change amount of resource occupancy rate within adjacent time windows, the fluctuation mode type of the candidate peak is judged. If the change amount of resource occupancy rate within multiple consecutive adjacent time windows is positive and gradually increases, it is the continuously rising type; if the change amount of resource occupancy rate alternates between positive and negative and fluctuates within a certain range, it is the oscillating fluctuation type; if the change amount of resource occupancy rate within a certain adjacent time window suddenly increases and is much higher than other time windows, it is the sudden spike type.

[0063] Step S4253: Determine the stability level of the candidate peak according to the fluctuation mode type; the stability level is used to characterize the persistence possibility of the candidate peak at future time points.

[0064] The stability level is determined according to the fluctuation mode type, and different fluctuation mode types correspond to different stability levels. The candidate peaks of the continuously rising type usually have a relatively high stability level because their rising trend is relatively stable; the stability level of the candidate peaks of the oscillating fluctuation type is relatively low because their fluctuation conditions are relatively complex; the stability level of the candidate peaks of the sudden spike type is the lowest because they appear suddenly and are difficult to predict. For example, a stability level range is set for each fluctuation mode type, and the stability level of the candidate peak is determined according to its fluctuation mode type.

[0065] Step S4254: If the fluctuation mode type of the candidate peak is the continuously rising type and the stability level is higher than the preset fluctuation threshold, perform dynamic adjustment on the candidate peak to generate an adjusted candidate peak; the dynamic adjustment includes reducing the numerical amplitude of the candidate peak according to the slope of the change amount of resource occupancy rate within adjacent time windows.

[0066] The preset fluctuation threshold is a threshold preset for judging the stability of the candidate peak. When the fluctuation mode type of the candidate peak is the continuously rising type and the stability level is higher than the preset fluctuation threshold, in order to avoid overestimating the resource occupancy rate, dynamic adjustment is performed on the candidate peak. The dynamic adjustment reduces the numerical amplitude of the candidate peak according to the slope of the change amount of resource occupancy rate within adjacent time windows. For example, if the slope of the change amount of resource occupancy rate within adjacent time windows is large, it means that the resource occupancy rate rises rapidly. At this time, appropriately reduce the numerical amplitude of the candidate peak to generate an adjusted candidate peak.

[0067] Step S4255: If the fluctuation pattern type of the candidate peak is the oscillatory fluctuation type and the stability level is lower than the preset fluctuation threshold, retain the original value of the candidate peak and mark it as a high-risk fluctuation area.

[0068] When the fluctuation pattern type of the candidate peak is the oscillatory fluctuation type and the stability level is lower than the preset fluctuation threshold, it indicates that the fluctuation of the candidate peak is relatively unstable and there is a greater risk. At this time, retain the original value of the candidate peak and mark it as a high-risk fluctuation area for special processing in subsequent resource allocation. For example, when allocating resources, more cautious evaluation and processing are carried out on the resource occupancy corresponding to the high-risk fluctuation area.

[0069] Step S4256: Generate the final predicted resource occupancy rate based on the adjusted candidate peaks, high-risk fluctuation areas, and unmarked candidate peaks; where the final predicted resource occupancy rate is the weighted average of the adjusted candidate peaks and the unmarked candidate peaks, and the candidate peaks corresponding to the high-risk fluctuation areas are given the lowest weight.

[0070] The final predicted resource occupancy rate is calculated comprehensively based on the adjusted candidate peaks, high-risk fluctuation areas, and unmarked candidate peaks. To reduce the impact of high-risk fluctuation areas on the prediction result, the candidate peaks corresponding to the high-risk fluctuation areas are given the lowest weight. For example, using the weighted average method, calculate the weighted average of the adjusted candidate peaks and the unmarked candidate peaks according to the weight of each candidate peak to obtain the final predicted resource occupancy rate.

[0071] Step S430: Determine the available resource capacity of the target resource node based on the predicted resource occupancy rate and the resource upper limit threshold in the execution constraint conditions.

[0072] The predicted resource occupancy rate represents the resource occupancy of the target resource node in a future period of time, and the resource upper limit threshold in the execution constraint conditions is a restriction on resource usage. By comparing the predicted resource occupancy rate and the resource upper limit threshold, determine the available resource capacity of the target resource node. For example, if the predicted resource occupancy rate is 80% and the resource upper limit threshold is 90%, then the available resource capacity of the target resource node is 10%.

[0073] Step S440: Generate a first resource allocation instruction based on the available resource capacity. The first resource allocation instruction includes a list of allocable resource types and a capacity upper limit.

[0074] The first resource allocation instruction is an instruction generated based on the available resource capacity of the target resource node, which specifies the list of allocable resource types and the upper limit of the capacity of each resource. For example, for a target resource node with computing resources and storage resources, according to the available resource capacity, a first resource allocation instruction is generated, which may include the list of types of allocable computing resources (such as the number of CPU cores) and storage resources (such as disk space) and the corresponding upper limits of capacity.

[0075] If the current traversal level node is associated with a task allocation model, the following steps are performed: Step S450: Extract the priority parameters that match the task priority feature set from the task execution request. The priority parameters include the task deadline, the number of dependent tasks, and the resource exclusivity flag.

[0076] The task priority feature set is a set of features used to measure the task priority. The priority parameters are the parameters extracted from the task execution request that match the task priority feature set. The task deadline represents the time point when the task must be completed. The number of dependent tasks represents the number of other tasks that the task depends on. The resource exclusivity flag represents whether the task needs to exclusively occupy certain resources. For example, for an execution request of a data processing task, the priority parameters such as the task deadline, the number of dependent tasks, and the resource exclusivity flag are extracted from it.

[0077] Step S460: Generate a task allocation weight according to the priority parameters.

[0078] The task allocation weight is a weight value calculated according to the priority parameters and used to measure the task priority. For example, the task urgency flag, the resource dependency parameter, and the task association parameter are parsed from the task execution request. The initial priority weight is determined according to the task urgency flag, and the initial priority weight is first adjusted based on the resource dependency parameter. The data sharing degree with other tasks is determined according to the task association parameter, and the adjusted initial priority weight is secondarily adjusted through the data sharing degree. Finally, the weight value after two adjustments is normalized and mapped to generate the task allocation weight. The normalization mapping scales the weight value proportionally to a preset weight interval so that the sum of the task allocation weights at the same level is a fixed value.

[0079] As an implementation manner, step S460, generating a task allocation weight according to the priority parameters, may specifically include: Step S461: Parse the task urgency flag, the resource dependency parameter, and the task association parameter from the task execution request.

[0080] The task urgency identifier is used to indicate the urgency of a task, the resource dependency parameter represents the degree of dependence of the task on resources, and the task correlation parameter represents the degree of association between the task and other tasks. These parameters are extracted by parsing the task execution request. For example, for the execution request of a software development task, the task urgency identifier (such as urgent, general, not urgent), the resource dependency parameter (such as the quantity of computing resources, storage resources, etc. required), and the task correlation parameter (such as the dependency relationship with other development tasks) are parsed out.

[0081] Step S462: Determine the initial priority weight according to the task urgency identifier, and perform a first adjustment on the initial priority weight based on the resource dependency parameter.

[0082] The initial priority weight is determined according to the task urgency identifier. The higher the urgency, the greater the initial priority weight. Then, a first adjustment is performed on the initial priority weight according to the resource dependency parameter. If the task has a high degree of dependence on resources, the initial priority weight is increased; otherwise, it is decreased. For example, for an urgent task, the initial priority weight is set to a relatively high value. If the task also has a high degree of dependence on resources, the weight value is appropriately increased on the basis of the initial priority weight.

[0083] Step S463: Determine the data sharing degree with other tasks according to the task correlation parameter, and perform a second adjustment on the adjusted initial priority weight through the data sharing degree.

[0084] The data sharing degree is determined according to the task correlation parameter, which represents the degree of data sharing between the task and other tasks. A second adjustment is performed on the adjusted initial priority weight through the data sharing degree. If the data sharing degree is high, the priority weight is increased to ensure data interaction and collaboration between tasks; otherwise, the priority weight is decreased. For example, for a task with a large amount of data sharing with other tasks, the weight value is appropriately increased on the basis of the adjusted initial priority weight.

[0085] Step S464: Perform a standardized mapping on the weight value after two adjustments to generate a task allocation weight; wherein, the standardized mapping includes scaling the weight value proportionally to a preset weight interval so that the sum of the task allocation weights at the same level is a fixed value.

[0086] The standardized mapping is to make the task allocation weights at the same level comparable and consistent. The weight value after two adjustments is scaled proportionally to a preset weight interval, so that the sum of the task allocation weights at the same level is a fixed value. For example, the min-max normalization method is used to map the weight value to the interval [0, 1], while ensuring that the sum of all task allocation weights at the same level is 1, to generate the task allocation weight.

[0087] Step S470: Generate a second resource allocation instruction based on the task assignment weights and the task parallelism limit in the execution constraint conditions. The second resource allocation instruction includes a task execution order identifier and a resource allocation ratio.

[0088] The task parallelism limit is a limit on the number of tasks executed simultaneously. Based on the task assignment weights and the task parallelism limit, determine the execution order and resource allocation ratio of the tasks, and generate a second resource allocation instruction. For example, for multiple tasks, sort the tasks according to the task assignment weights, and in combination with the task parallelism limit, determine which tasks can be executed simultaneously and the resource ratio allocated to each task, and generate a second resource allocation instruction including the task execution order identifier and the resource allocation ratio.

[0089] Step S500: According to the resource allocation instruction, dynamically allocate the computing task corresponding to the task execution request to the target resource node, and trigger the target resource node to execute the computing task.

[0090] The resource allocation instruction clarifies information such as the resource type and quantity required for the task and the task execution order. According to these instructions, the device dynamically allocates the computing task corresponding to the task execution request to the appropriate target resource node. Dynamic allocation means that the device makes flexible adjustments according to the real-time state of the resources and the requirements of the tasks to ensure the efficient use of resources. The target resource node refers to the resource node selected to execute the computing task. When the task allocation is completed, the device triggers the target resource node to start executing the computing task. For example, for a database query task, the device allocates the task to a target resource node with sufficient storage resources and computing power according to the resource allocation instruction, and then sends an execution instruction to the target resource node to trigger it to start executing the database query task.

[0091] As an implementation manner, the method provided by the embodiment of the present invention may further include the following steps: Step S600: During the process of the target resource node executing the computing task, collect the actual resource occupancy data and task execution progress data of the target resource node in real time.

[0092] The actual resource occupancy data reflects the actual resource usage of the target resource node when executing the computing task, such as the processor load rate, memory occupancy rate, etc. The task execution progress data represents the execution progress of the computing task, such as the completed task ratio, remaining task quantity, etc. By collecting these data in real time, the execution situation of the task and the resource usage situation can be understood in a timely manner. For example, use monitoring tools to collect the actual resource occupancy data such as the processor load rate and memory occupancy rate of the target resource node in real time, and at the same time obtain the task execution progress data from the task management device.

[0093] Step S700: Generate a model optimization instruction for the resource prediction model based on the deviation value between the actual resource occupancy data and the predicted resource occupancy rate.

[0094] The deviation value is the difference between the actual resource occupancy data and the predicted resource occupancy rate, which reflects the prediction error of the resource prediction model. Generating a model optimization instruction for the resource prediction model based on the deviation value is used to optimize and adjust the resource prediction model to improve the prediction accuracy of the model. For example, if the actual resource occupancy rate is significantly higher than the predicted resource occupancy rate, it indicates that the model underestimates the resource demand, and a corresponding model optimization instruction is generated to adjust the parameters and structure of the model.

[0095] Step S800: Generate a parameter update instruction for the task allocation model based on the task execution progress data and the expected progress threshold in the execution constraint conditions.

[0096] The expected progress threshold is the expected value of the task execution progress specified in the execution constraint conditions. By comparing the task execution progress data with the expected progress threshold, it is judged whether the task is executed as planned. If there is a deviation between the task execution progress and the expected progress threshold, a parameter update instruction for the task allocation model is generated to update the parameters of the task allocation model to optimize the task allocation scheme. For example, if the task execution progress is significantly behind the expected progress threshold, it indicates that the task allocation is unreasonable, and a parameter update instruction is generated to adjust the parameters of the task allocation model and re-allocate the tasks.

[0097] Step S900: Based on the model optimization instruction and the parameter update instruction, perform incremental training on the associated resource prediction model and task allocation model in the resource scheduling topology graph to generate an updated resource scheduling topology graph.

[0098] Incremental training refers to further training and optimizing the model based on the original model using new data. Based on the model optimization instruction and parameter update instruction, incremental training is performed on the associated resource prediction model and task allocation model in the resource scheduling topology graph to improve the performance and adaptability of the model. For example, the actual resource occupancy data is merged with the historical resource dataset to generate an extended resource dataset, and noise filtering processing is performed on the extended resource dataset. The new dynamic occupancy feature set is extracted from the extended resource dataset, and the new dynamic occupancy feature set is input into the online learning layer of the resource prediction model. The convolutional kernel parameters of the resource prediction model are updated through gradient backpropagation in the online learning layer, while the parameters of the long short-term memory network layer are frozen. The task execution progress data is compared with the preset progress deviation threshold. If the deviation exceeds the threshold, the node to be split is located in the decision tree layer of the task allocation model, and based on the task association degree parameter in the task execution progress data, the feature re-partitioning of the node to be split is performed to generate an updated task allocation model. Finally, an updated resource scheduling topology graph is generated according to the updated model.

[0099] Step S1000: When a new task execution request is received, resource allocation is performed using the updated resource scheduling topology graph.

[0100] When the device receives a new task execution request, the updated resource scheduling topology graph is used for resource allocation to improve the accuracy and efficiency of resource allocation. The associated models in the updated resource scheduling topology graph have undergone incremental training and can better adapt to the actual situation of resource nodes and the requirements of tasks, thus achieving more reasonable resource allocation. For example, for a new computing task execution request, according to the updated resource scheduling topology graph, by traversing the hierarchical nodes, the associated resource prediction model and task allocation model are called to generate a resource allocation instruction, and the task is allocated to the appropriate target resource node.

[0101] As an implementation, in step S900, the incremental training of the associated resource prediction model and task allocation model in the resource scheduling topology graph may specifically include: Step S910: Merge the actual resource occupancy data with the historical resource dataset to generate an extended resource dataset, and perform noise filtering processing on the extended resource dataset; The actual resource occupancy data is the data collected in real time during the execution of the computing task on the target resource node. It is merged with the historical resource dataset to obtain an extended resource dataset. The noise filtering processing is to remove the noise and outliers in the extended resource dataset and improve the data quality. For example, statistical methods (such as mean filtering, median filtering, etc.) are used to perform noise filtering processing on the extended resource dataset to remove the abnormal fluctuations and noise points in the data.

[0102] Step S920: Extract the newly added dynamic occupancy feature set from the extended resource dataset, and input the newly added dynamic occupancy feature set into the online learning layer of the resource prediction model; The newly added dynamic occupancy feature set is a feature set matching the resource demand type extracted from the extended resource dataset, which contains new resource occupancy information. Inputting the newly added dynamic occupancy feature set into the online learning layer of the resource prediction model allows the model to learn and adjust based on the new data. For example, use a method similar to Step S410 to extract the newly added dynamic occupancy feature set from the extended resource dataset and input it into the online learning layer of the resource prediction model.

[0103] Step S930: Update the convolution kernel parameters of the resource prediction model through gradient backpropagation in the online learning layer, and freeze the parameters of the long short-term memory network layer at the same time; Gradient backpropagation is an optimization algorithm for updating model parameters. By calculating the gradient of the loss function with respect to the model parameters, the model parameters are updated to reduce the value of the loss function. During incremental training, the convolution kernel parameters of the resource prediction model are updated through gradient backpropagation in the online learning layer to adapt to the new data. At the same time, to maintain the stability of the model, the parameters of the long short-term memory network layer are frozen and not updated. For example, use the stochastic gradient descent (SGD) algorithm to update the convolution kernel parameters of the resource prediction model through gradient backpropagation.

[0104] Step S940: Compare the task execution progress data with a preset progress deviation threshold. If the deviation exceeds the threshold, locate the node to be split in the decision tree layer of the task allocation model; The preset progress deviation threshold is a threshold preset for judging whether the task execution progress is normal. Comparing the task execution progress data with this threshold, if the deviation exceeds the threshold, it indicates that the parameters of the task allocation model may need to be adjusted. Locate the node to be split in the decision tree layer of the task allocation model. The node to be split is the node in the decision tree that needs to be further divided. For example, use the decision tree algorithm to construct the task allocation model. When the task execution progress deviation exceeds the threshold, find the node that needs to be split from the nodes of the decision tree.

[0105] Step S950: Based on the task association degree parameter in the task execution progress data, re-partition the features of the node to be split to generate an updated task allocation model.

[0106] The task correlation parameter represents the degree of association between a task and other tasks. Based on the task correlation parameter, the to-be-split node is re-partitioned in terms of features, and the partitioning rule of the node is re-determined to improve the accuracy of the task allocation model. For example, according to information such as the data sharing degree and dependency relationship in the task correlation parameter, the to-be-split node is re-partitioned in terms of features to generate an updated task allocation model.

[0107] As an implementation manner, the method provided by the embodiment of the present invention may further include the following steps: Step S1100: When it is detected that there is a conflict in the resource allocation instructions of multiple task execution requests, obtain a conflict resolution model from the conflict handling nodes of the resource scheduling topology graph; A conflict in resource allocation instructions means that there is a contradiction or competition between the resource allocation instructions of multiple task execution requests. For example, multiple tasks request the same resource. A conflict handling node is a node in the resource scheduling topology graph specifically used to handle conflicts, and a conflict resolution model is a model used to resolve conflicts in resource allocation instructions. When a conflict is detected, the conflict resolution model is obtained from the conflict handling nodes. For example, during the resource scheduling process, the resource allocation instructions of multiple task execution requests are monitored in real time. When a conflict is found, a pre-trained conflict resolution model is obtained from the conflict handling nodes of the resource scheduling topology graph.

[0108] Step S1200: Use the conflict resolution model to evaluate the conflict impact degree on the priority parameters, resource requirement types, and historical execution records of multiple task execution requests; Evaluating the conflict impact degree means evaluating the impact degree of the conflict on task execution and resource allocation. The conflict resolution model evaluates the conflict impact degree based on the priority parameters, resource requirement types, and historical execution records of multiple task execution requests. For example, extract the task type label, resource occupation duration, and dependent resource list from each task execution request, determine the business domain to which the task belongs according to the task type label, and obtain the historical conflict resolution strategy of the business domain. Based on the historical conflict resolution strategy and the resource occupation duration, calculate the resource release cost of each task execution request. According to the overlap degree of the dependent resource lists, determine the resource competition intensity between tasks. Integrate the resource release cost and the resource competition intensity to generate the conflict level score and recommended solution in the conflict impact degree evaluation result.

[0109] As an implementation manner, in step S1200, using the conflict resolution model to evaluate the conflict impact degree on the priority parameters, resource requirement types, and historical execution records of multiple task execution requests may specifically include: Step S1210: Extract the task type label, resource occupation duration, and dependent resource list from each task execution request; The task type tag is used to identify the type of task, such as a computing task, a storage task, etc.; the resource occupation duration represents the length of time the task occupies resources; the dependent resource list lists the resources that the task depends on. Extract this information from each task execution request to provide data support for subsequent conflict impact assessment. For example, for an execution request of a data processing task, extract the task type tag (such as a data processing task), the resource occupation duration (such as the expected execution time), and the dependent resource list (such as the required computing resources, storage resources, etc.).

[0110] Step S1220: Determine the business domain to which the task belongs according to the task type tag, and obtain the historical conflict resolution strategy of the business domain; The business domain refers to the business area to which the task belongs. Different business domains may have different conflict resolution strategies. Determine the business domain to which the task belongs according to the task type tag, and then obtain the historical conflict resolution strategy of this business domain from the database or knowledge base. For example, if the task type tag is "financial transaction task", then determine that the business domain to which the task belongs is the financial field, and obtain the historical conflict resolution strategy from the knowledge base of the financial field.

[0111] Step S1230: Calculate the resource release cost of each task execution request based on the historical conflict resolution strategy and the resource occupation duration; The resource release cost refers to the cost brought by releasing the resources occupied by the task, including costs such as task interruption and data loss. Calculate the resource release cost of each task execution request based on the historical conflict resolution strategy and the resource occupation duration. For example, according to the resource release cost evaluation method for different types of tasks in the historical conflict resolution strategy, combined with the resource occupation duration of the task, calculate the resource release cost of each task execution request.

[0112] Step S1240: Determine the resource competition intensity between tasks according to the overlap degree of the dependent resource list; The overlap degree of the dependent resource list represents the degree of demand for the same resources by multiple tasks. The higher the overlap degree, the greater the resource competition intensity. Determine the resource competition intensity between tasks by comparing the dependent resource lists of multiple tasks. For example, calculate the intersection of the dependent resource lists of multiple tasks, and determine the resource competition intensity according to the size of the intersection.

[0113] Step S1250: Synthesize the resource release cost and the resource competition intensity to generate the conflict level score and recommended solution in the conflict impact assessment result.

[0114] The conflict level score is a quantitative assessment of the severity of a conflict, and the recommended solution is a suggestion for resolving the conflict based on the conflict level score and the specific situation of the task. By integrating the resource release cost and the resource competition intensity, the conflict level score and the recommended solution in the conflict impact assessment result are generated. For example, using the weighted average method, the resource release cost and the resource competition intensity are comprehensively calculated to obtain the conflict level score. Based on the conflict level score and information such as the priority of the task, recommended solutions such as resource preemption and task queuing are given.

[0115] Step S1300: Generate a resource preemption instruction or a task queuing instruction according to the conflict impact assessment result; among them, the resource preemption instruction is used to forcibly release the resources occupied by low-priority tasks, and the task queuing instruction is used to add the conflicting tasks to the waiting queue and allocate resources in order. According to the conflict impact assessment result, judge the severity of the conflict and the priority of the task, and generate corresponding instructions. If the conflict is relatively severe and low-priority tasks occupy critical resources, a resource preemption instruction is generated to forcibly release the resources occupied by low-priority tasks to meet the needs of high-priority tasks; if the conflict is not very severe, the conflicting tasks can be added to the waiting queue and resources are allocated in order, and at this time a task queuing instruction is generated. For example, for a situation with a high conflict level score, a resource preemption instruction is generated to forcibly release the resources occupied by low-priority tasks; for a situation with a low conflict level score, a task queuing instruction is generated to add the conflicting tasks to the waiting queue.

[0116] Step S1400: Feedback the resource preemption instruction or the task queuing instruction to the target resource node, and update the task status flag in the resource scheduling topology graph.

[0117] Feedback the generated resource preemption instruction or task queuing instruction to the target resource node, so that the target resource node performs the corresponding operation. At the same time, update the task status flag in the resource scheduling topology graph to reflect the latest status of the task. For example, send the resource preemption instruction to the target resource node. After receiving the instruction, the target resource node releases the resources occupied by low-priority tasks; send the task queuing instruction to the target resource node, and the target resource node adds the conflicting tasks to the waiting queue. At the same time, update the status flag of the task in the resource scheduling topology graph, such as updating the status flag of the low-priority task to "resource preempted" and updating the status flag of the conflicting task to "queuing".

[0118] As an implementation manner, the method provided by the embodiment of the present invention may further include the following steps: Step S1500: When initially constructing the resource scheduling topology graph, collect the hardware configuration data, network topology structure and historical task execution logs of multiple resource nodes.

[0119] The hardware configuration data reflects the hardware performance of resource nodes, such as processor model, memory capacity, disk capacity, etc.; the network topology describes the network connection relationships among resource nodes; the historical task execution logs record the detailed information of tasks executed by resource nodes in the past. When initially constructing the resource scheduling topology graph, collect this data to provide basic data for subsequent clustering analysis and model training of resource nodes. For example, obtain the hardware configuration data by querying the hardware management device of the resource node, use a network topology discovery tool to obtain the network topology, and obtain the historical task execution logs from the task management device.

[0120] Step S1600: Extract the computing power characteristics, storage capacity characteristics, and network bandwidth characteristics of resource nodes from the hardware configuration data.

[0121] The computing power characteristics represent the computing power of resource nodes, such as the number of cores and main frequency of the processor, etc.; the storage capacity characteristics represent the storage capacity of resource nodes, such as disk capacity, memory capacity, etc.; the network bandwidth characteristics represent the network transmission ability of resource nodes, such as network bandwidth, transmission rate, etc. Extract these characteristics from the hardware configuration data for subsequent clustering analysis. For example, extract the number of cores and main frequency of the processor from the hardware configuration data as the computing power characteristics, extract the disk capacity and memory capacity as the storage capacity characteristics, and extract the network bandwidth and transmission rate as the network bandwidth characteristics.

[0122] Step S1700: Extract the task execution duration characteristics, resource utilization characteristics, and failed task distribution characteristics from the historical task execution logs.

[0123] The task execution duration characteristics represent the time length of task execution, the resource utilization characteristics represent the resource usage efficiency of resource nodes during task execution, and the failed task distribution characteristics represent the distribution of failed tasks across different resource nodes and time. Extract these characteristics from the historical task execution logs for subsequent clustering analysis. For example, count the execution time of each task as the task execution duration characteristics, calculate the processor utilization rate, memory utilization rate, etc. of resource nodes during task execution as the resource utilization characteristics, and analyze the distribution of failed tasks across different resource nodes and time as the failed task distribution characteristics.

[0124] Step S1800: Based on the computing power characteristics, storage capacity characteristics, network bandwidth characteristics, task execution duration characteristics, resource utilization characteristics, and failed task distribution characteristics, perform clustering analysis on resource nodes to generate the initial hierarchical node division result.

[0125] Cluster analysis is a method of grouping data objects. Based on computing power characteristics, storage capacity characteristics, network bandwidth characteristics, task execution duration characteristics, resource utilization rate characteristics, and failed task distribution characteristics, cluster analysis is performed on resource nodes. Resource nodes with similar characteristics are grouped into the same set to generate the initial hierarchical node division result. For example, using the K-means clustering algorithm, the similarity between resource nodes is calculated based on these characteristics, and resource nodes with higher similarity are grouped into the same set to obtain the initial hierarchical node division result.

[0126] Step S1900: According to the initial hierarchical node division result, assign an initial resource prediction model or task assignment model to each hierarchical node, and generate a resource scheduling topology map through historical data playback training.

[0127] According to the initial hierarchical node division result, assign an initial resource prediction model or task assignment model to each hierarchical node. The resource prediction model is used to predict the future resource occupancy of resource nodes, and the task assignment model is used to determine the task assignment scheme. Through historical data playback training, these models are trained and optimized using historical data to generate a resource scheduling topology map. For example, assign an initial resource prediction model and task assignment model to each hierarchical node, use historical data such as historical task execution logs and hardware configuration data to train the models, adjust the parameters of the models, and finally generate a resource scheduling topology map.

[0128] In summary, the multi-line collaborative scheduling method based on artificial intelligence provided by the present invention realizes dynamic resource allocation and scheduling of tasks by constructing a resource scheduling topology map and combining a resource prediction model and a task assignment model. At the same time, by incrementally training the models with real-time collected data and handling resource allocation instruction conflicts, the accuracy and efficiency of resource allocation are improved, and it can better adapt to the requirements of different tasks and the actual situation of resource nodes.

[0129] Please refer to Figure 2 , Figure 2Schematic structural diagram of a multi-line collaborative scheduling device provided by an embodiment of the present invention. The multi-line collaborative scheduling device at least includes a processor 101, a communication interface 102, and a memory 103. Among them, the processor 101, the communication interface 102, and the memory 103 can be connected through a bus or other means. Among them, the processor 101 (or Central Processing Unit (CPU)) is the computing core and control core of the multi-line collaborative scheduling device, which can parse various instructions in the multi-line collaborative scheduling device and process various data of the multi-line collaborative scheduling device. The communication interface 102 can optionally include standard wired interfaces, wireless interfaces (such as WI-FI, mobile communication interfaces, etc.), and can be used to send and receive data under the control of the processor 101; the communication interface 102 can also be used for the transmission and interaction of internal data of the multi-line collaborative scheduling device. The memory 103 (Memory) is the memory device in the multi-line collaborative scheduling device, used to store programs and data. It can be understood that the memory 103 here can include both the built-in memory of the multi-line collaborative scheduling device and, of course, the extended memory supported by the multi-line collaborative scheduling device. The memory 103 provides a storage space, and this storage space stores the operating system of the multi-line collaborative scheduling device, which can include but is not limited to: Android system, iOS system, Windows Phone system, etc., and the present invention does not make any limitations in this regard.

[0130] In one embodiment, the processor 101 executes the multi-line collaborative scheduling method based on artificial intelligence provided above in the embodiments of the present invention by running the computer program in the memory 103.

Claims

1. A multi-line collaborative scheduling method based on artificial intelligence, characterized in that: The method comprises: Obtain a resource scheduling topology map and receive a task execution request; each hierarchical node of the resource scheduling topology map is associated with a resource prediction model or a task allocation model, and in each execution path of the resource scheduling topology map, the hierarchical node associated with the task allocation model is located after the hierarchical node associated with the resource prediction model; wherein the resource prediction model is a resource prediction model generated based on resource occupancy feature set training, and the task allocation model is a task allocation model generated based on task priority feature set; According to the task requirement parameters carried in the task execution request, parsing the resource requirement type and execution constraint conditions corresponding to the task execution request; Starting from the starting level node of the resource scheduling topology graph, traverse all level nodes in the resource scheduling topology graph to obtain a resource prediction model or a task allocation model associated with the currently traversed level node; Based on the resource requirement type and the execution constraint, a resource allocation instruction corresponding to the currently traversed level node is generated through a resource prediction model or a task allocation model associated with the currently traversed level node; According to the resource allocation instruction, the computing task corresponding to the task execution request is dynamically allocated to the target resource node, and the target resource node is triggered to execute the computing task.

2. The method according to claim 1, characterized in that The generating, based on the resource requirement type and the execution constraint, a resource allocation instruction corresponding to the currently traversed level node through a resource prediction model or a task allocation model associated with the currently traversed level node, comprises: If the currently traversed level node is associated with a resource prediction model, the following steps are performed: Extracting a dynamic occupancy feature set matching the resource demand type from a historical resource data set of the resource scheduling topology map; Performing real-time input adaptation on the resource prediction model according to the dynamic occupancy feature set to generate a predicted resource occupancy rate of the target resource node; Determining the available resource capacity of the target resource node based on the predicted resource occupancy rate and the resource upper limit threshold in the execution constraint condition; generating a first resource allocation instruction according to the available resource capacity, wherein the first resource allocation instruction includes a list of allocatable resource types and a capacity upper limit; If the currently traversed level node is associated with a task allocation model, the following steps are performed: Extracting priority parameters matching the task priority feature set from the task execution request, the priority parameters including task deadline, number of dependent tasks and resource exclusivity identifier; generating a task allocation weight according to the priority parameter; Based on the task allocation weight and the task parallelism limit in the execution constraint condition, a second resource allocation instruction is generated, where the second resource allocation instruction includes a task execution sequence identifier and a resource allocation ratio.

3. The method according to claim 2, characterized in that The step of extracting a dynamic occupancy feature set matching the resource demand type from a historical resource data set of the resource scheduling topology graph includes: Obtaining resource occupancy curves of multiple resource nodes recorded in the historical resource data set within a preset time period; Perform feature segmentation processing on each resource occupancy curve, and extract the peak fluctuation characteristics, continuous occupancy duration characteristics, and load balancing characteristics corresponding to each segment of the curve; According to the resource category identifier in the resource demand type, a target feature subset associated with the resource category identifier is selected from the peak fluctuation feature, the continuous occupancy duration feature and the load balancing feature; The target feature subset is normalized to generate the dynamic occupancy feature set; wherein the normalization includes mapping feature values ​​of different dimensions to a uniform numerical interval and eliminating linear correlations between features.

4. The method according to claim 2, characterized in that: The step of adapting the resource prediction model to real-time input according to the dynamic occupancy feature set to generate the predicted resource occupancy rate of the target resource node includes: Inputting the dynamic occupancy feature set into the feature encoding layer of the resource prediction model to generate a high-dimensional feature vector; Through the time series analysis layer of the resource prediction model, the high-dimensional feature vector is segmented into sliding windows to obtain feature fragments corresponding to multiple local time windows; Performing convolution kernel extraction processing on each feature segment to generate a convolution feature map of each time window, and inputting the convolution feature map into the long short-term memory network layer of the resource prediction model; Modeling the temporal dependency of the convolutional feature graph through the long short-term memory network layer, and outputting a resource occupancy prediction sequence of the target resource node at a future time point; The predicted resource occupancy rate is determined according to a maximum value and a fluctuation trend in the resource occupancy rate prediction sequence.

5. The method according to claim 2, characterized in that: Generating the task allocation weight according to the priority parameter includes: Parsing the task urgency indicator, resource dependency parameter and task relevance parameter from the task execution request; Determining an initial priority weight according to the task urgency identifier, and performing a first adjustment on the initial priority weight based on the resource dependency parameter; Determine the data sharing degree with other tasks according to the task association parameter, and perform a second adjustment on the adjusted initial priority weight according to the data sharing degree; The weight values ​​adjusted twice are subjected to standard mapping to generate the task allocation weights; wherein the standard mapping includes scaling the weight values ​​to a preset weight range in proportion so that the sum of the task allocation weights at the same level is a fixed value.

6. The method according to claim 1, characterized in that The method further comprises: In the process of the target resource node executing the computing task, real-time collection of actual resource occupancy data and task execution progress data of the target resource node; Generate a model optimization instruction for a resource prediction model according to a deviation value between the actual resource occupancy data and the predicted resource occupancy rate; Generate a parameter update instruction of a task allocation model according to the task execution progress data and the expected progress threshold in the execution constraint condition; Based on the model optimization instructions and parameter update instructions, incremental training is performed on the resource prediction model and the task allocation model associated with the resource scheduling topology map to generate an updated resource scheduling topology map; When a new task execution request is received, resource allocation is performed using the updated resource scheduling topology map; The incremental training of the resource prediction model and the task allocation model associated with the resource scheduling topology graph includes: Merging the actual resource occupancy data with the historical resource data set to generate an extended resource data set, and performing noise filtering on the extended resource data set; Extracting a newly added dynamic occupancy feature set from the extended resource data set, and inputting the newly added dynamic occupancy feature set into the online learning layer of the resource prediction model; Performing gradient back-propagation update on the convolution kernel parameters of the resource prediction model through the online learning layer, and freezing the parameters of the long short-term memory network layer; Comparing the task execution progress data with a preset progress deviation threshold, and if the deviation exceeds the threshold, locating the node to be split from the decision tree layer of the task allocation model; Based on the task correlation parameters in the task execution progress data, the features of the nodes to be split are redivided to generate an updated task allocation model.

7. The method according to claim 1, characterized in that The method further comprises: When it is detected that a conflict exists in resource allocation instructions of multiple task execution requests, a conflict resolution model is obtained from a conflict processing node of the resource scheduling topology graph; Performing conflict impact assessment on priority parameters, resource requirement types and historical execution records of the plurality of task execution requests through the conflict resolution model; Generate a resource preemption instruction or a task queuing instruction according to the conflict impact assessment result; wherein the resource preemption instruction is used to forcibly release the resources occupied by low-priority tasks, and the task queuing instruction is used to add the conflicting tasks to the waiting queue and allocate resources in order; The resource preemption instruction or task queuing instruction is fed back to the target resource node, and the task status identifier in the resource scheduling topology diagram is updated.

8. The method according to claim 7, characterized in that The conflict resolution model is used to evaluate the conflict impact of the priority parameters, resource requirement types and historical execution records of the plurality of task execution requests, including: Extract the task type label, resource occupancy time, and dependent resource list from each task execution request; Determine the business domain to which the task belongs according to the task type tag, and obtain a historical conflict resolution strategy for the business domain; Calculate the resource release cost of each task execution request based on the historical conflict resolution strategy and resource occupation duration; Determining the intensity of resource competition between tasks according to the degree of overlap of the dependent resource lists; The resource release cost and resource competition intensity are comprehensively considered to generate a conflict level score and a recommended solution in the conflict impact assessment result.

9. The method according to claim 1, characterized in that: The method further comprises: When initially constructing the resource scheduling topology map, hardware configuration data, network topology structure and historical task execution logs of multiple resource nodes are collected; Extracting computing power characteristics, storage capacity characteristics and network bandwidth characteristics of resource nodes from the hardware configuration data; Extracting task execution time characteristics, resource utilization characteristics, and failed task distribution characteristics from the historical task execution log; Based on the computing power characteristics, storage capacity characteristics, network bandwidth characteristics, task execution time characteristics, resource utilization characteristics and failed task distribution characteristics, cluster analysis is performed on resource nodes to generate initial level node division results; According to the initial hierarchical node division result, an initial resource prediction model or task allocation model is allocated to each hierarchical node, and the resource scheduling topology map is generated through historical data playback training.

10. A multi-line collaborative scheduling device, characterized in that: include: a memory, wherein a computer program is stored in the memory; A processor, used to load the computer program to implement the multi-line collaborative scheduling method based on artificial intelligence as described in any one of claims 1-9.

Images