A heterogeneous computing resource unified management, control and scheduling system

Abstract

The application discloses a heterogeneous computing resource unified management and control scheduling system, relates to the technical field of heterogeneous computing resource scheduling, and solves the technical problems that the existing scheduling system is difficult to perceive the semantic difference of reasoning requests, is also difficult to measure the semantic quality of reasoning output, and lacks a fine migration recovery mechanism driven by output quality; the system comprises a semantic analysis module, a computing power link arrangement module, and an SLA semantic guarantee module; the semantic analysis module extracts cognitive complexity features of reasoning requests in real time through a lightweight semantic router; the computing power link arrangement module decomposes reasoning tasks into a subtask sequence with a dependency relationship according to the cognitive complexity features, matches target computing power types, and allocates the target computing power types to corresponding nodes in a heterogeneous resource pool, and generates a global resource allocation scheme according to the dependency relationship; the SLA semantic guarantee module defines a semantic-level SLA commitment comprising a reasoning accuracy loss threshold value and a logical coherence score threshold value according to the cognitive complexity features, and triggers cross-node migration or model switching when the threshold values are not met.

Description

Technical Field

[0001] This invention belongs to the field of heterogeneous computing resource scheduling technology, specifically relating to a unified management and scheduling system for heterogeneous computing resources. Background Technology

[0002] With the widespread application of large-scale language models and multimodal models, AI inference services have placed higher demands on the scheduling and management of heterogeneous computing resources (GPUs, CPUs, FPGAs, and DCUs, etc.).

[0003] Existing heterogeneous computing power scheduling systems typically allocate resources and balance loads based on the hardware resource requirements of tasks (e.g., number of CPU cores, GPU memory capacity, floating-point arithmetic capability). Their scheduling decisions are primarily based on resource monitoring data and historical load statistics. However, this approach has the following shortcomings in multi-model hybrid deployment scenarios: First, different inference requests exhibit significant differences in logical complexity, context dependency length, and result determinism requirements. These differences directly affect the computational path, resource consumption, and execution quality of model inference. However, existing scheduling systems lack methods to extract these semantic features from the request text, leading to scheduling decisions that are inconsistent with the actual request. First, there is a disconnect between the actual execution requirements and the actual needs. Second, traditional SLAs use system performance indicators such as response time and throughput as constraints, while in AI inference scenarios, users are actually concerned with semantic quality such as the accuracy and logical consistency of the inference results. When the scheduling system cannot perceive the degradation of semantic quality, even if the SLA meets the requirements at the system performance level, the user experience may still not be guaranteed. Third, when the quality of inference output deteriorates due to resource contention or hardware differences among allocated nodes, the system finds it difficult to migrate problem subtasks to suitable nodes under the constraints of task dependencies, and it is also difficult to avoid the waste of computing power caused by repeated execution failures after migration.

[0004] Therefore, there is an urgent need for a heterogeneous computing resource scheduling system that can perceive the semantic features of reasoning requests, ensure output quality, and has fine-grained migration and recovery capabilities. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art; to this end, this invention proposes a unified management and scheduling system for heterogeneous computing resources, which is used to solve the technical problems that existing scheduling systems have difficulty in perceiving the semantic differences of inference requests, in measuring the semantic quality of inference outputs, and lacking a fine-grained migration and recovery mechanism driven by output quality.

[0006] To address the above problems, this invention provides a unified management and scheduling system for heterogeneous computing resources, comprising the following modules:

[0007] Semantic parsing module: Used to receive reasoning requests and extract the cognitive complexity features of the requests in real time through a lightweight semantic router, including context length requirements, logical reasoning depth and result determinism requirements;

[0008] Computing power link orchestration module: used to decompose the reasoning task into a sequence of sub-tasks with dependencies based on the cognitive complexity characteristics, and match the corresponding target computing power type for each sub-task. The computing power link orchestration module allocates each sub-task to the corresponding node in the heterogeneous resource pool through dynamic load balancing, and forms a global resource allocation scheme based on the dependencies.

[0009] SLA Semantic Guarantee Module: Used to receive the cognitive complexity features to define a semantic-level SLA commitment, which includes a reasoning accuracy loss threshold and a logical coherence score threshold;

[0010] During task execution, the SLA semantic guarantee module collects the actual output of each subtask and compares the logical coherence score and the inference accuracy loss value with the corresponding SLA thresholds. When any threshold is not met, the SLA semantic guarantee module sends a migration instruction to the computing power link orchestration module.

[0011] After receiving the migration instruction, the computing power link orchestration module reallocates the subtasks to be migrated to the new node. The SLA semantic guarantee module collects the output quality again after the new node has completed the execution for judgment. During cross-node migration or model switching, the heterogeneous resource pool maintains the synchronization of task status through cache consistency overhead suppression mechanism and asynchronous communication mechanism, and reclaims the resources of abnormal nodes through memory leak detection and memory fragmentation suppression strategy.

[0012] Preferably, in the semantic parsing module, the cognitive complexity features of the request are extracted in real time through a lightweight semantic router, wherein:

[0013] The inference request is input into a semantic encoder built on the Transformer encoder. The semantic encoder is deployed after end-to-end training for a cognitive complexity feature extraction task and outputs an encoded representation containing context semantic vectors and an attention weight matrix.

[0014] The context length requirement is determined by statistically analyzing the distribution span of the number of times each input token is attended in the attention weight matrix, and then dividing the distribution span by the total number of input tokens.

[0015] The inference request is matched with a preset set of logical connectors, and the ratio of the number of matches to the total number of input tokens is used as the logical connector density. The nesting level of each conditional clause in the inference request is analyzed, and the maximum nesting level is used as the recursion depth of the conditional clause. The weighted sum of the logical connector density and the recursion depth is used as the logical inference depth.

[0016] Input the probability distribution of the output vocabulary corresponding to the reasoning request into the entropy calculation function to obtain the probability distribution entropy value; use the reciprocal of the probability distribution entropy value as the result certainty requirement.

[0017] Preferably, in the computing power link orchestration module, the reasoning task is decomposed into a sequence of sub-tasks with dependencies based on the cognitive complexity characteristics, including:

[0018] Obtain the structure configuration file of the target large model corresponding to the inference request. The structure configuration file contains several modal processing branches of the target large model, as well as the sub-modules and inter-layer connection relationships within each branch.

[0019] Calculate the probability threshold based on the logical reasoning depth value in the cognitive complexity feature. ,in, The preset scaling factor is used. This represents the depth of logical reasoning.

[0020] Along the forward propagation path when executing the inference request in the target large model, the inference task is divided into several modal-level subtasks with each modal processing branch of the target large model as the first-level segmentation boundary. For each modal-level subtask, the activation probability value of each sub-module contained in the modal-level subtask in the forward propagation path is traversed with the sub-module and inter-layer connection relationship as the second-level segmentation boundary. Sub-modules with activation probability values ​​< probability threshold are removed, and the remaining continuous sub-module sequence is taken as the atomic-level subtask of the inference task.

[0021] Extracting the input tensor set of each atomic-level subtask With output tensor set For the first The atomic-level subtask and the first Atom-level subtasks, when and At that time, the judgment of the first The atomic-level subtask is the first The prerequisite tasks for each atomic-level subtask; when and At that time, the judgment of the first The atomic-level subtask and the first There are no dependencies between the atomic-level subtasks, and they can be executed in parallel. Traverse all pairs of atomic-level subtasks to generate a dependency directed graph with each atomic-level subtask as a vertex and the aforementioned prerequisite dependencies as directed edges.

[0022] Preferably, in the computing power link orchestration module, matching the corresponding target computing power type for each subtask includes:

[0023] Obtain the baseline performance vector of each node in the heterogeneous resource pool. The baseline performance vector includes the throughput per unit time of each node in four dimensions: floating-point operation, integer operation, matrix multiplication and addition operation, and logical branch operation.

[0024] For any atomic-level subtask, extract the computational feature vector of the sub-module corresponding to the atomic-level subtask, which includes the computational proportion of the sub-module in the four dimensions;

[0025] The calculated feature vector is multiplied by the baseline performance vector of each node to obtain the estimated execution efficiency score of the atomic subtask on each node.

[0026] Using the result certainty requirement value in the cognitive complexity feature as a weighting coefficient, the estimated execution efficiency score on each node is multiplied by the weighting coefficient, and the computing power type corresponding to the node with the largest product is selected as the target computing power type of the atomic-level subtask.

[0027] Preferably, the computing power link orchestration module allocates each subtask to the corresponding node in the heterogeneous resource pool through dynamic load balancing, including:

[0028] The instantaneous load status vector of each node in the heterogeneous resource pool is collected in real time. The instantaneous load status vector includes the current occupancy rate of each type of resource and the current number of concurrent tasks of each node.

[0029] For any atomic-level subtask with a selected target computing power type, nodes whose resource occupancy rate for each type is less than a preset occupancy threshold are selected from the heterogeneous resource pool to form a candidate node set.

[0030] If the candidate node set is empty, calculate the L2 norm of the resource occupancy vector of each node and the cosine similarity between the resource occupancy vector and the calculated feature vector of the atomic subtask. Use the weighted sum of the negative of the L2 norm and the cosine similarity as the matching score of the node. Sort all nodes in descending order of matching score and add the top N nodes to the candidate node set until the candidate node set is not empty, where N is a preset expansion coefficient.

[0031] From the candidate node set, the node with the lowest current task concurrency is selected as the allocation target; if multiple nodes have the same concurrency, the L2 norm of each node's resource utilization vector is used as the load score, and the node with the lowest load score is selected as the allocation target. Preferably, the following is true:

[0032] Preferably, in the computing power link orchestration module, forming a global resource allocation scheme based on the dependencies includes:

[0033] Using each atomic-level subtask as a vertex and the directed edges in the dependent directed graph as constraints, a task execution directed acyclic graph corresponding to the inference task is constructed.

[0034] Perform critical path analysis on the directed acyclic graph for the task, and calculate the longest path length from any vertex to the exit vertex as the critical path length of the vertex.

[0035] Based on the critical path length of each vertex in descending order, the vertices in the directed acyclic graph of the task are topologically sorted to generate an execution priority sequence for each atomic subtask.

[0036] Based on the allocation results of each atomic-level subtask and the execution priority sequence, a global resource allocation scheme is generated, which includes the binding relationship between each atomic-level subtask and the target node, the execution priority of each atomic-level subtask, and the cross-node data transmission path.

[0037] Preferably, the SLA semantic guarantee module defines a semantic-level SLA commitment, including:

[0038] The feature values ​​of each dimension in the cognitive complexity feature are compared with their respective preset thresholds, and the logical coherence score threshold and reasoning accuracy loss threshold in the semantic-level SLA commitment are determined based on the comparison results.

[0039] The logical reasoning depth value is compared with a preset depth threshold. When the logical reasoning depth value is greater than the preset depth threshold, the logical coherence score threshold is set as a first coherence threshold; otherwise, it is set as a second coherence threshold. The result certainty requirement value is compared with a preset certainty threshold. When the result certainty requirement value is less than the preset certainty threshold, the reasoning accuracy loss threshold is set as a first loss threshold; otherwise, it is set as a second loss threshold. The context length requirement value is compared with a preset length threshold. When the context length requirement value is greater than the preset length threshold, the logical coherence score threshold is increased by a first correction amount, and the reasoning accuracy loss threshold is decreased by a second correction amount.

[0040] Preferably, in the SLA semantic guarantee module, collecting the actual output of each subtask and triggering cross-node migration or model switching includes:

[0041] After each atomic-level subtask is completed, the output quality evaluator deployed in the execution node corresponding to the atomic-level subtask is invoked. The output quality evaluator includes a semantic similarity calculation submodule and a logical consistency detection submodule.

[0042] The semantic similarity calculation submodule inputs the actual output sequence of the atomic-level subtask and the preset reference output sequence into the pre-trained semantic matching model to obtain the semantic similarity score output by the semantic matching model. The difference between 1 and the semantic similarity score is used as the reasoning accuracy loss value. The preset reference output sequence is obtained as follows: for reasoning requests with a standard answer, the standard answer is retrieved from the knowledge base as the preset reference output sequence; for reasoning requests without a standard answer, multiple output samples are generated through multiple samplings, and the average semantic similarity between the multiple output samples is used as the reasoning accuracy loss value.

[0043] The logical consistency detection submodule performs factual consistency scoring, reasoning chain integrity scoring, and conclusion self-consistency scoring on the actual output sequence, and uses the weighted sum of the three scores as the logical coherence score.

[0044] When the logical coherence score is detected to be less than the logical coherence score threshold or the inference accuracy loss value is greater than the inference accuracy loss threshold, the SLA semantic guarantee module sends a migration instruction to the computing power link orchestration module. The migration instruction carries the atomic-level subtask identifier to be migrated and the current execution stage status identifier.

[0045] After receiving the migration instruction, the computing power link orchestration module queries all downstream atomic subtasks in the dependency directed graph that have the atomic subtask to be migrated as a prerequisite, suspends the execution of the downstream atomic subtasks, unbinds the atomic subtask to be migrated from the current allocation node, and reallocates the atomic subtask to be migrated to a new node by obtaining the instantaneous load state vector of each node in the heterogeneous resource pool, filtering the candidate node set, and selecting the node with the smallest current task concurrency as the allocation target.

[0046] After the atomic-level subtask to be migrated is reassigned to a new node, the computing power link orchestration module obtains the Euclidean distance between the resource utilization vector of the new node and the computational feature vector of the atomic-level subtask to be migrated. When the Euclidean distance is greater than a preset distance threshold, the SLA semantic guarantee module directly triggers model switching and routes the current inference request to the backup model execution engine. When the Euclidean distance is less than or equal to the preset distance threshold, the atomic-level subtask to be migrated is executed on the new node, and the output quality is collected again. If the semantic-level SLA commitment is still not met, the SLA semantic guarantee module triggers model switching.

[0047] The beneficial effects of this invention are:

[0048] This invention extracts cognitive complexity features from reasoning requests in real time through a semantic parsing module, extending the decision-making basis of the scheduling system from hardware resource specifications to the semantic level of the requests. This solves the problem that existing scheduling systems cannot perceive the differences in requests and achieves dynamic adaptation of scheduling granularity to the cognitive complexity of requests.

[0049] This invention extends the service level commitment from system performance indicators such as response time and throughput to semantic quality indicators such as inference accuracy loss and logical coherence score through the SLA semantic guarantee module. It also uses a multi-dimensional linkage threshold setting mechanism to match the SLA commitment with the actual quality requirements of the request, thus solving the problem that existing SLAs cannot measure the semantic quality of inference.

[0050] This invention introduces a two-level dynamic segmentation mechanism in task decomposition, dynamically adjusts the probability threshold based on the logical reasoning depth to control the granularity of sub-module retention, so that high-complexity requests retain the complete reasoning chain to ensure quality, and low-complexity requests eliminate redundant sub-modules to improve efficiency, thus achieving an on-demand balance between accuracy and efficiency.

[0051] This invention first predicts the matching degree between the new node and the subtask using Euclidean distance after migration is triggered. The task is executed only on the new node if the matching degree is sufficient. If the matching degree is still insufficient after execution, the backup model is switched. If the matching degree is insufficient, the backup model is switched directly without further execution attempts. This invention achieves the filtering out of unsuitable nodes by predicting the matching degree, avoiding the waste of computing power caused by repeated execution attempts across multiple nodes, and shortening the recovery time in scenarios of quality degradation. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the module flow of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Please see Figure 1 As shown, this invention is a unified management and scheduling system for heterogeneous computing resources, comprising the following modules:

[0055] Semantic parsing module: Used to receive reasoning requests and extract the cognitive complexity features of the requests in real time through a lightweight semantic router, including context length requirements, logical reasoning depth and result determinism requirements;

[0056] Computing power link orchestration module: used to decompose the reasoning task into a sequence of sub-tasks with dependencies based on the cognitive complexity characteristics, and match the corresponding target computing power type for each sub-task. The computing power link orchestration module allocates each sub-task to the corresponding node in the heterogeneous resource pool through dynamic load balancing, and forms a global resource allocation scheme based on the dependencies.

[0057] SLA Semantic Guarantee Module: Used to receive the cognitive complexity features to define a semantic-level SLA commitment, which includes a reasoning accuracy loss threshold and a logical coherence score threshold;

[0058] During task execution, the SLA semantic guarantee module collects the actual output of each subtask and compares the logical coherence score and the inference accuracy loss value with the corresponding SLA thresholds. When any threshold is not met, the SLA semantic guarantee module sends a migration instruction to the computing power link orchestration module.

[0059] After receiving the migration instruction, the computing power link orchestration module reallocates the subtasks to be migrated to the new node. The SLA semantic guarantee module collects the output quality again after the new node has completed the execution for judgment. During cross-node migration or model switching, the heterogeneous resource pool maintains the synchronization of task status through cache consistency overhead suppression mechanism and asynchronous communication mechanism, and reclaims the resources of abnormal nodes through memory leak detection and memory fragmentation suppression strategy.

[0060] In one embodiment of the present invention, the semantic parsing module extracts the cognitive complexity features of the request in real time through a lightweight semantic router, including:

[0061] The inference request is input into a semantic encoder built on the Transformer encoder. The semantic encoder is deployed after end-to-end training for a cognitive complexity feature extraction task and outputs an encoded representation containing context semantic vectors and an attention weight matrix.

[0062] The context length requirement is determined by statistically analyzing the distribution span of the number of times each input token is attended in the attention weight matrix, and then dividing the distribution span by the total number of input tokens.

[0063] The inference request is matched with a preset set of logical connectors, and the ratio of the number of matches to the total number of input tokens is used as the logical connector density. The nesting level of each conditional clause in the inference request is analyzed, and the maximum nesting level is used as the recursion depth of the conditional clause. The weighted sum of the logical connector density and the recursion depth is used as the logical inference depth.

[0064] Input the probability distribution of the output vocabulary corresponding to the reasoning request into the entropy calculation function to obtain the probability distribution entropy value; use the reciprocal of the probability distribution entropy value as the result certainty requirement.

[0065] Specifically, the semantic encoder adopts a Transformer encoder architecture. The encoder is pre-trained before model deployment. The training data uses text samples covering various types, including question answering, dialogue, document summarization, and logical reasoning. Each training sample is labeled with three dimensions: context length requirement level (long / medium / short), logical reasoning depth level (high / medium / low), and result certainty level (certain / medium / uncertain). During training, the text of the training samples is input into the encoder to obtain the encoded representation and attention weight matrix. Three parallel classification heads are connected at the top of the encoder to predict the levels of the three dimensions. The loss function is the weighted sum of the cross-entropy losses of the three classification heads. ,in, , and The cross-entropy loss is calculated for three classification heads: context length requirement, logical reasoning depth, and result certainty requirement. , and The weighting coefficients are determined using the analytic hierarchy process (AHP). In this embodiment, The value is 0.5. The value is 0.3. The value is 0.2; training stops when the accuracy of the three classifications on the validation set stabilizes and no longer improves, and the model weights are saved. During deployment, the three classification heads are removed, only the encoder part is retained, and it runs in lightweight inference mode, without gradient calculation, only performing forward propagation;

[0066] After tokenization, the original text of the inference request is input into the semantic encoder. The semantic encoder performs multi-head self-attention computation and outputs the encoded representation and attention weight matrix. The dimension of the attention weight matrix is... ,in, For the number of attention heads, Given the total number of input tokens, the system averages the weights of all attention heads along the head dimension, resulting in a dimension of... The average attention weight matrix, the matrix in which the first... Line number The elements of the column represent the first... indivual For the first indivual The system iterates through each column of the average attention weight matrix and counts the number of elements in that column whose values ​​exceed a preset attention threshold (0.01 in a preferred embodiment), which is then used as the attention level. The number of times it receives attention, in various The maximum minus the minimum among the number of times the data received attention is used as the distribution span value. This distribution span value is then divided by the input. total The resulting ratio serves as the context length requirement;

[0067] The system maintains a preset set of logical connectors, which is constructed by collecting common logical connectors and categorizing them by function as follows: Causal connectors: because, therefore, thus, due to, so, thereby, etc.; Adversative connectors: but, however, however, yet, although, although, etc.; Conditional connectors: if, supposing, as long as, unless, if, etc.; Progressive connectors: moreover, and even, moreover, etc.; Parallel connectors: and, with, and, as well as, and simultaneously, etc.; The set contains no fewer than 50 logical connector items. The system matches the original text of the reasoning request against the set and counts the total number of matched logical connectors. ,by Divide by input total The resulting ratio is used as the logical connector density. The system performs dependency parsing on the original text of the inference request, identifying clause structures such as conditional adverbial clauses, concessive adverbial clauses, and causal adverbial clauses. The parsing employs a pre-trained dependency parsing model trained on Treebank corpus, which outputs the dependency relationships between words and clause boundaries. The system counts the nesting level of each conditional clause, using the maximum nesting level as the recursion depth. The system then performs a weighted sum of the logical connector density and the recursion depth of the conditional clause to obtain the logical inference depth. : ,in, For logical connector density, For the recursion depth of the conditional clause, and The weighting coefficients are determined using the analytic hierarchy process (AHP). In this embodiment, The value is 0.6. The value is 0.4;

[0068] The system obtains the word probability distribution of the target large model corresponding to the inference request in the output layer, that is, the distribution of the probability values ​​of each candidate word output by the softmax layer. ,in, The system inputs the probability distribution into the entropy calculation function, where the vocabulary size is specified. , ,in, The entropy value of the probability distribution. For the first The probability value of each word, if Then set The reciprocal of the probability distribution entropy is used as the requirement for the certainty of the result. : , A higher value indicates a higher requirement for the accuracy of the output.

[0069] In one embodiment of the present invention, the computing power link orchestration module decomposes the reasoning task into a sequence of sub-tasks with dependencies based on the cognitive complexity characteristics, including:

[0070] Obtain the structure configuration file of the target large model corresponding to the inference request. The structure configuration file contains several modal processing branches of the target large model, as well as the sub-modules and inter-layer connection relationships within each branch.

[0071] Calculate the probability threshold based on the logical reasoning depth value in the cognitive complexity feature. ,in, The preset scaling factor is used. This represents the depth of logical reasoning.

[0072] Along the forward propagation path when executing the inference request in the target large model, the inference task is divided into several modal-level subtasks with each modal processing branch of the target large model as the first-level segmentation boundary. For each modal-level subtask, the activation probability value of each sub-module contained in the modal-level subtask in the forward propagation path is traversed with the sub-module and inter-layer connection relationship as the second-level segmentation boundary. Sub-modules with activation probability values ​​< probability threshold are removed, and the remaining continuous sub-module sequence is taken as the atomic-level subtask of the inference task.

[0073] Extracting the input tensor set of each atomic-level subtask With output tensor set For the first The atomic-level subtask and the first Atom-level subtasks, when and At that time, the judgment of the first The atomic-level subtask is the first The prerequisite tasks for each atomic-level subtask; when and At that time, the judgment of the first The atomic-level subtask and the first There are no dependencies between the atomic-level subtasks, and they can be executed in parallel. Traverse all pairs of atomic-level subtasks to generate a dependency directed graph with each atomic-level subtask as a vertex and the aforementioned prerequisite dependencies as directed edges.

[0074] Specifically, the system obtains the structure configuration file of the target large model corresponding to the inference request. This configuration file is provided by the model developer during model deployment and is stored in a structured data format (e.g., JSON or Protobuf), containing complete topological information of the model. If the model developer does not provide a structure configuration file, the system automatically generates the configuration file through static analysis of the model computation graph. The specific method of static analysis is as follows: the system loads the pre-trained weight file of the target large model, calls the model parsing interface of the deep learning framework (e.g., PyTorch or TensorFlow), traverses all named modules of the model, and for each module, the system records its type (e.g., nn.Linear, nn.Conv2d, nn.MultiheadAttention, nn.LayerNorm, etc.), input tensor specifications (shape, data type), output tensor specifications, and the branch path it belongs to (e.g., image encoding branch, text encoding branch, cross-modal fusion branch). After traversal, the system groups the modules according to the branch path, and the modules under the same branch are arranged in the forward propagation order to generate the structure configuration file. The system then uses the logical inference depth value in the cognitive complexity feature to generate the structure configuration file. Calculate probability threshold ,in, A preset scaling factor is used to control the sensitivity of the mapping between logical reasoning depth and probability thresholds. During system initialization, optimization experiments are conducted using a set of verification requests (containing several requests each of simple, medium, and complex logic requests). The optimal balance between the number of sub-modules retained after task decomposition and the quality of execution results is selected. The optimal value;

[0075] The system performs a two-level segmentation along the forward propagation path when executing inference requests from the target large model: The first level segmentation uses the modal processing branches of the target large model as boundaries. Taking the visual-language multimodal large model as an example, the target large model typically includes three modal processing branches: visual encoding branch, text encoding branch, and multimodal fusion branch. The system uses the starting endpoint of the data flow in the forward propagation path as the segmentation point, dividing the inference task into three modal-level subtasks: visual encoding modal-level subtask, text encoding modal-level subtask, and multimodal fusion modal-level subtask. The second level segmentation is performed within each modal-level subtask. For each modal-level subtask, the system uses the sub-modules and inter-layer connections contained in that modal-level subtask as boundaries, traversing the activation probability values ​​of each sub-module contained in that modal-level subtask in the forward propagation path. The activation probability values ​​are obtained by inserting lightweight monitoring points at the output of each layer of the model. The monitoring points record the L2 norm of the output tensor of that layer. During the execution of historical inference requests, if the L2 norm of the output tensor of a certain layer is greater than the preset activation threshold, then that layer is marked as "activated" in that request; otherwise, it is marked as "inactive." The system calculates the activation frequency (number of activations / total number of requests) of each layer in the most recent N inference requests, and uses this frequency as the activation probability value of that submodule. For the first inference request, the system uses the average activation statistics of each layer provided during the model development phase as the initial value. The system compares the activation probability value of each submodule with the probability threshold. Compare them one by one, and eliminate those with an activation probability value less than 1. The sub-modules retain the sequence of consecutive sub-modules as atomic subtasks. If a sub-module is removed, its input tensor is directly passed to the first sub-module that is retained, without changing the overall direction of the data flow.

[0076] The system extracts the input tensor sets of each atomic-level subtask. With output tensor set Each tensor set uses tensor names as its elements. These tensor names are obtained from the model's computation graph and are globally unique. For the... The atomic-level subtask and the first Atom-level subtasks ( The system executes the following decision logic: Calculate and The intersection: If When, it indicates the first A tensor output by each subtask is directly used as the first... The input tensor of each subtask is used to determine the first... The atomic-level subtask is the first The prerequisite tasks for each atomic-level subtask; if and This indicates that there is no tensor transitive relationship between the two subtasks, and it is determined that the two are not dependent on each other and can be executed in parallel on different nodes. The system traverses all atomic-level subtask pairs, performs the above determination on each pair, and finally generates a dependency directed graph with each atomic-level subtask as the vertex and the preceding dependency relationship obtained by the determination as the directed edge.

[0077] In one embodiment of the present invention, the computing power link orchestration module matches the corresponding target computing power type for each subtask, including:

[0078] Obtain the baseline performance vector of each node in the heterogeneous resource pool. The baseline performance vector includes the throughput per unit time of each node in four dimensions: floating-point operation, integer operation, matrix multiplication and addition operation, and logical branch operation.

[0079] For any atomic-level subtask, extract the computational feature vector of the sub-module corresponding to the atomic-level subtask, which includes the computational proportion of the sub-module in the four dimensions;

[0080] The calculated feature vector is multiplied by the baseline performance vector of each node to obtain the estimated execution efficiency score of the atomic subtask on each node.

[0081] Using the result certainty requirement value in the cognitive complexity feature as a weighting coefficient, the estimated execution efficiency score on each node is multiplied by the weighting coefficient, and the computing power type corresponding to the node with the largest product is selected as the target computing power type of the atomic-level subtask.

[0082] Specifically, during the initialization phase, the system performs performance benchmark tests on each node, and the benchmark performance vector for each node is represented as follows: ,in, This refers to the throughput per unit time in the floating-point operation dimension. The throughput per unit time in integer operations. This represents the throughput per unit time in the dimension of matrix multiplication and addition operations. This represents the throughput per unit time in the logical branch operation dimension. The throughput per unit time for each dimension is obtained as follows: Floating-point operation throughput is obtained by running large-scale matrix multiplication (e.g., 1024×1024 FP32 matrix multiplication) for 60 seconds, and the total number of floating-point operations is divided by the test duration; Integer operation throughput is obtained by running integer matrix multiplication (e.g., 2048×2048 INT8 matrix multiplication) for 60 seconds, and the total number of integer operations is divided by the test duration; Matrix multiplication-addition operation throughput is obtained by running matrix multiplication with intensive multiplication-addition instructions for 60 seconds, and the number of multiplication-addition operations is divided by the test duration; Logical branch operation throughput is obtained by running a benchmark program containing numerous conditional branches for 60 seconds, and the number of branch operations is divided by the test duration. Tests are performed when nodes are idle to avoid interference from normal tasks; the system automatically re-executes the benchmark every 24 hours or when node hardware configuration changes.

[0083] For any atomic-level subtask, the system extracts the computational feature vector of the corresponding submodule. The computational feature vector is represented as: ,in, This represents the percentage of computational cost in floating-point operations. This represents the percentage of computational cost for integer operations. This represents the percentage of computational cost in matrix multiplication and addition operations. This represents the percentage of computational cost in logical branch operations. The specific extraction method for this percentage is as follows: The system obtains the computation graph representation of this submodule during forward propagation. The computation graph is a directed acyclic graph, with nodes representing operators (e.g., convolution, matrix multiplication, activation functions, normalization, conditional branching, etc.) and edges representing tensor flow directions. The system traverses each operator node in the computation graph and categorizes them into one of four computational dimensions based on operator type: convolution operators, matrix multiplication operators, and vector dot product operators are categorized into the matrix multiplication-addition operation dimension; activation function operators (e.g., ReLU, Sigmoid, GELU) and normalization operators (e.g., LayerNorm, BatchNorm) are categorized into floating-point operations. Dimensions: Data movement operators (e.g., tensor transpose, remodeling) and comparison operators are categorized under the integer operation dimension; conditional branch operators (e.g., if-else, select) and control flow operators are categorized under the logical branch operation dimension. For each operator, the system obtains the number of floating-point operations (FLOPs) of that operator from the model configuration file as a computational cost metric. If the FLOPs of an operator cannot be directly obtained (e.g., control flow operators), the system uses the average execution time of that operator under standard input as a substitute value for the computational cost metric. The system calculates the total computational cost of operators under each dimension, divides the total computational cost of each dimension by the total computational cost of the four dimensions, and obtains the computational cost percentage of each dimension.

[0084] The system will compute the feature vectors of atomic-level subtasks. The baseline performance vector of each node Perform a dot product operation to obtain the atomic-level subtask at the node. The estimated execution efficiency score : The baseline performance vector reflects the node's supply capacity for each type of computation. The higher the dot product of the two, the better the node's computational capacity matches the computational requirements of the subtask.

[0085] The system uses the result certainty requirement value in the cognitive complexity feature. As a weighting coefficient From the entropy value of the probability distribution Directly determine, A higher value indicates greater uncertainty in the model's output to the request, requiring higher-quality computing resources to ensure inference quality; the estimated execution efficiency is scored across each node. Multiply Receive a weighted score : The system selects The computing power type corresponding to the largest node is taken as the target computing power type for this atomic-level subtask. When the efficiency is high, the system prioritizes the node with the highest estimated execution efficiency score to ensure inference quality; when... When the weighted scores of each node are lower, the gap between them narrows, allowing the system to flexibly choose between multiple computing power types during the subsequent load balancing phase.

[0086] In one embodiment of the present invention, the computing power link orchestration module allocates each subtask to the corresponding node in the heterogeneous resource pool through dynamic load balancing:

[0087] The instantaneous load status vector of each node in the heterogeneous resource pool is collected in real time. The instantaneous load status vector includes the current occupancy rate of each type of resource and the current number of concurrent tasks of each node.

[0088] For any atomic-level subtask with a selected target computing power type, nodes whose resource occupancy rate for each type is less than a preset occupancy threshold are selected from the heterogeneous resource pool to form a candidate node set.

[0089] If the candidate node set is empty, calculate the L2 norm of the resource occupancy vector of each node and the cosine similarity between the resource occupancy vector and the calculated feature vector of the atomic subtask. Use the weighted sum of the negative of the L2 norm and the cosine similarity as the matching score of the node. Sort all nodes in descending order of matching score and add the top N nodes to the candidate node set until the candidate node set is not empty, where N is a preset expansion coefficient.

[0090] From the candidate node set, the node with the smallest current task concurrency is selected as the allocation target; if there are multiple nodes with the same concurrency, the L2 norm of the resource utilization vector of each node is used as the load score, and the node with the smallest load score is selected as the allocation target.

[0091] Specifically, the system collects the instantaneous load state vectors of each node in the heterogeneous resource pool in real time through resource monitoring agents deployed on each node. The instantaneous load state vector of each node is represented as follows: ,in, Number the nodes. Sampling time, This refers to the utilization rate of the floating-point arithmetic unit. This refers to the occupancy rate of integer arithmetic units. This refers to the occupancy rate of matrix multiplication and addition units. This refers to the occupancy rate of the logical branch processing unit. This represents the current task concurrency. The resource utilization rates for each type are obtained as follows: for GPU nodes, real-time utilization of each computing unit is obtained by calling the GPU management API (e.g., NVIDIA Management Library); for CPU nodes, utilization of each core is obtained by reading the operating system's / proc / stat file; for FPGA nodes, logic resource utilization is obtained by reading the FPGA management interface. All utilization rates are percentage values. Current task concurrency. The total number of subtasks currently being executed on a node is obtained by statistically analyzing the active task list maintained by the task scheduler on the node. The resource monitoring agent collects load data once every preset sampling period (preferably set to 500ms in this embodiment), encapsulates the collected instantaneous load status vector into a heartbeat message and reports it to the central scheduler. If a node fails to report a heartbeat within three consecutive sampling periods, the central scheduler marks the node as unavailable.

[0092] For any atomic-level subtask with a selected target computing power type, the system selects nodes from the heterogeneous resource pool whose resource utilization rate for each type is less than a preset utilization threshold, forming a candidate node set. : ,in, To preset the occupancy threshold, when the occupancy rate of all resource types on a node is below 80%, the node still has sufficient remaining resources to process new tasks. If the occupancy rate of any resource exceeds 80%, the node is in a high-load state, and continuing to allocate new tasks may cause that resource to become a performance bottleneck. The specific operation of the filtering process is as follows: the system traverses all nodes in the resource pool whose computing power type matches the target computing power type, and checks the instantaneous load state vector of each node. The occupancy rate of the first four dimensions, if all dimensions are less than If so, then the node is included in the candidate node set;

[0093] When the candidate node set When empty, the system calculates the matching score between each node and the atomic-level subtask. ,in, For nodes The resource utilization vector, for L2 norm, These are the feature vectors of atomic-level subtasks. for and cosine similarity, and The weighting coefficients are determined using the analytic hierarchy process (AHP). The value is 0.6. The value is 0.4; Match score This reflects the overall load level and the degree to which the load type matches the computing requirements of the nodes. A larger value indicates that the node is more suitable as an allocation target. The system follows the... Sort all nodes in descending order, and add the top N nodes to the candidate node set.

[0094] From the candidate node set In the middle, the system selects the current task concurrency. The smallest node is selected as the allocation target, and the selection rule is as follows: This rule has the highest priority, prioritizing the node with the fewest concurrent tasks to balance the task load across nodes and prevent some nodes from being overloaded while others are idle. If multiple nodes have the same number of concurrent tasks q_j(t), the system calculates the resource utilization vector for each node. The L2 norm, the formula for calculating the L2 norm is: The L2 norm reflects the overall load level of a node across various resource types; a smaller norm indicates a lower overall load on all resource types. The system selects the node with the smallest L2 norm as the allocation target. ,in, This selection strategy ensures that, with the same number of concurrent connections, the system prioritizes the node with the lowest overall load, minimizing the possibility of a single resource becoming a bottleneck.

[0095] In one embodiment of the present invention, the computing power link orchestration module forms a global resource allocation scheme based on the dependency relationship, including:

[0096] Using each atomic-level subtask as a vertex and the directed edges in the dependent directed graph as constraints, a task execution directed acyclic graph corresponding to the inference task is constructed.

[0097] Perform critical path analysis on the directed acyclic graph for the task, and calculate the longest path length from any vertex to the exit vertex as the critical path length of the vertex.

[0098] Based on the critical path length of each vertex in descending order, the vertices in the directed acyclic graph of the task are topologically sorted to generate an execution priority sequence for each atomic subtask.

[0099] Based on the allocation results of each atomic-level subtask and the execution priority sequence, a global resource allocation scheme is generated, which includes the binding relationship between each atomic-level subtask and the target node, the execution priority of each atomic-level subtask, and the cross-node data transmission path.

[0100] Specifically, the system constructs a directed acyclic graph of task execution corresponding to the inference task, using each atomic-level subtask as a vertex and the directed edges in the dependency directed graph as constraints. ,in, Given a set of vertices, each vertex corresponds to an atomic-level subtask. The task execution directed acyclic graph (DAG) is a set of directed edges, with each edge pointing from the preceding dependent task to the subsequent task. The DAG is constructed as follows: traverse all vertices and edges in the dependent directed graph, copy the vertices to vertices in the DAG, and copy the directed edges to directed edges in the DAG. Since the dependent directed graph itself is a DAG (the forward propagation path of the inference task does not have cyclic dependencies), the resulting graph is the task execution DAG. For each vertex, the system labels the estimated execution time. The estimated execution time is obtained by estimating the number of operators contained in the subtask and the size of the input data. Specifically, it is done by iterating through each operator contained in the subtask and summing the estimated execution time of each operator on the target computing power type. The estimated execution time of each operator on the target computing power type is obtained through benchmark testing.

[0101] The system performs critical path analysis on the directed acyclic graph (DAG) of task execution. The critical path is defined as the longest path from the entry vertex (a vertex with no incoming edges) to the exit vertex (a vertex with no outgoing edges), where the path length is the sum of the estimated execution times of all vertices along the path. The system uses a backward dynamic programming algorithm to calculate the critical path length for each vertex. The calculation process is as follows: For the set of exit vertices... ,in, Let be the out-degree of the vertex, if ,express If there are no subsequent tasks, it is the endpoint of the execution path; the critical path length is the estimated execution time for that vertex. For non-exit vertices, the formula for calculating the critical path length is: ,in, As vertex The estimated execution time, express There is a direction A directed edge, through a vertex The longest path length to the exit vertex is equal to The execution time of the vertex itself plus the maximum critical path length among all its successor vertices;

[0102] The system performs a topological sort based on the critical path length of each vertex, from largest to smallest. The specific sorting rule is as follows: first, sort all vertices by critical path length. Sort in descending order. Vertices with larger values ​​are listed first; for Vertices with the same value are arranged according to the topological order in the dependent directed graph (i.e., preceding dependent tasks are placed before subsequent tasks); after sorting, the system outputs each vertex in sequence according to the sorting result, generating the execution priority sequence of each atomic subtask. The execution priority sequence means that, in the event of resource contention, vertices with longer critical path lengths have higher execution priority. This ordering ensures that subtasks on the critical path receive computing resources first, avoiding delays in overall task completion time due to waiting for resources.

[0103] The system generates a global resource allocation scheme based on the allocation results and execution priority sequence of each atomic-level subtask. This global resource allocation scheme comprises the following three parts: Binding relationships: a mapping table between each atomic-level subtask and the target node, specifically in the form of: ,in, Atomic-level subtask identifier, The target node identifier for allocation; execution priority: the execution order number of each atomic-level subtask. Specifically, each subtask is assigned an integer value. , A smaller number indicates a higher execution priority. Execution priority numbers are assigned according to the execution priority sequence. The order in the allocation starts from 1 and increases sequentially; cross-node data transmission path: for any two subtasks that have a dependency relationship. and ( for (prerequisite dependent tasks), if Assigned to a node , Assigned to a node ,and If cross-node data transmission exists between the two, the system generates a transmission path for each pair of cross-node data transmissions. The generation method for cross-node data transmission paths is as follows: the system pre-obtains the network topology between nodes in the heterogeneous resource pool, including the network latency and available bandwidth between any two nodes. For tensor data to be transmitted, the system calculates the size of the data to be transmitted based on the tensor specifications (shape, data type). The system then retrieves the data from the source node... To the target node Among all available network paths, the path that meets the bandwidth constraints and has the shortest transmission time is selected as the data transmission path. The transmission time is estimated as follows: estimated transmission time = data volume / effective bandwidth of the path + path delay. The system encapsulates the binding relationship, execution priority and cross-node data transmission path into a global resource allocation scheme and distributes it to each execution node and the task scheduler of each node for execution.

[0104] In one embodiment of the present invention, the SLA semantic guarantee module defines a semantic-level SLA commitment, including:

[0105] The feature values ​​of each dimension in the cognitive complexity feature are compared with their respective preset thresholds, and the logical coherence score threshold and reasoning accuracy loss threshold in the semantic-level SLA commitment are determined based on the comparison results.

[0106] The logical reasoning depth value is compared with a preset depth threshold. When the logical reasoning depth value is greater than the preset depth threshold, the logical coherence score threshold is set as a first coherence threshold; otherwise, it is set as a second coherence threshold. The result certainty requirement value is compared with a preset certainty threshold. When the result certainty requirement value is less than the preset certainty threshold, the reasoning accuracy loss threshold is set as a first loss threshold; otherwise, it is set as a second loss threshold. The context length requirement value is compared with a preset length threshold. When the context length requirement value is greater than the preset length threshold, the logical coherence score threshold is increased by a first correction amount, and the reasoning accuracy loss threshold is decreased by a second correction amount.

[0107] Specifically, the semantic-level SLA commitment defined by the SLA semantic guarantee module is a binary constraint condition, which includes a logical coherence score threshold. and inference accuracy loss threshold With two independent thresholds, the final SLA commitment is expressed as: logical coherence score ≥ And the inference accuracy loss value is ≤ ;

[0108] The system will use the logical reasoning depth value from the cognitive complexity features. Compared with the preset depth threshold The comparison is as follows: A preset depth threshold is set by collecting inference request samples covering three logical levels (simple, medium, and complex) during the system initialization phase. For each sample, its logical inference depth value is calculated. The logical reasoning depth distribution of the samples was statistically analyzed, and the median of the distribution was used as a benchmark, combined with expert experience to determine the depth. The value of ; when When the request logic chain is long and requires multiple steps of reasoning, the system has higher requirements for the logical coherence of the output, and the logical coherence score threshold is set as the first coherence threshold. ;when When the request logic is relatively simple, the system sets the logical coherence score threshold to the second coherence threshold. ;in, and The system selects a batch of standard test requests with known logical complexity and runs them under different thresholds. The criterion for determining the optimal correlation between user satisfaction scores (manually labeled in the test environment) and the threshold settings is... and The value;

[0109] The system will use the result certainty requirement value from the cognitive complexity feature. With preset deterministic threshold The comparison involves: setting a deterministic threshold. By systematically collecting reasoning request samples covering both deterministic questions (such as mathematical calculations and fact-finding questions) and uncertain questions (such as creative writing and open-ended questions), the deterministic requirement value of the result for each sample is calculated. Statistical analysis of two types of samples The distribution of values, with the intersection of the two distributions as... The value of ; when When the request is highly uncertain and multiple reasonable answers exist, the system has a high tolerance for accuracy loss, and the inference accuracy loss threshold is set as the first loss threshold. ;when When the request is highly certain and the answer is clear, the system has stricter requirements for accuracy, and the inference accuracy loss threshold is set as the second loss threshold. ;in, and The system was tested under different loss thresholds using standard test requests. The acceptability of the inference results (assessed by domain experts) was used as the evaluation metric, and the maximum loss tolerance value with an acceptability rate of no less than 95% was selected as the maximum acceptable value. The minimum loss requirement value with an acceptable rate of not less than 99% is selected as... ;

[0110] The system will use the context length requirement value from the cognitive complexity features. With preset length threshold The comparison specifically involves: a preset length threshold. The distribution of context length requirement value L in historical inference requests was statistically analyzed using the system, with the 75th quantile of the distribution as the threshold. The value of ; when This indicates that the request requires attention to a longer context. In this scenario, it is more difficult to maintain the coherence of reasoning, and accuracy is more easily affected by noise in long sequences. Therefore, the system performs a double correction on the SLA constraint: increasing the logical coherence score threshold by a first correction amount. (For higher consistency requirements) the inference accuracy loss threshold is lowered by a second correction amount. (Requires stricter accuracy); when At this time, the system does not perform a correction operation; where the first correction amount With the second correction amount By collecting long context through the system ( ) and short context ( Two types of requests were tested, and the SLA compliance rate and user satisfaction were tested under different correction amounts. The combination of correction amounts that minimized the difference in satisfaction between long context requests and short context requests was selected.

[0111] In one embodiment of the present invention, the SLA semantic guarantee module collects the actual output of each subtask and triggers cross-node migration or model switching, including:

[0112] After each atomic-level subtask is completed, the output quality evaluator deployed in the execution node corresponding to the atomic-level subtask is invoked. The output quality evaluator includes a semantic similarity calculation submodule and a logical consistency detection submodule.

[0113] The semantic similarity calculation submodule inputs the actual output sequence of the atomic-level subtask and the preset reference output sequence into the pre-trained semantic matching model to obtain the semantic similarity score output by the semantic matching model. The difference between 1 and the semantic similarity score is used as the reasoning accuracy loss value. The preset reference output sequence is obtained as follows: for reasoning requests with a standard answer, the standard answer is retrieved from the knowledge base as the preset reference output sequence; for reasoning requests without a standard answer, multiple output samples are generated through multiple samplings, and the average semantic similarity between the multiple output samples is used as the reasoning accuracy loss value.

[0114] The logical consistency detection submodule performs factual consistency scoring, reasoning chain integrity scoring, and conclusion self-consistency scoring on the actual output sequence, and uses the weighted sum of the three scores as the logical coherence score.

[0115] When the logical coherence score is detected to be less than the logical coherence score threshold or the inference accuracy loss value is greater than the inference accuracy loss threshold, the SLA semantic guarantee module sends a migration instruction to the computing power link orchestration module. The migration instruction carries the atomic-level subtask identifier to be migrated and the current execution stage status identifier.

[0116] After receiving the migration instruction, the computing power link orchestration module queries all downstream atomic subtasks in the dependency directed graph that have the atomic subtask to be migrated as a prerequisite, suspends the execution of the downstream atomic subtasks, unbinds the atomic subtask to be migrated from the current allocation node, and reallocates the atomic subtask to be migrated to a new node by obtaining the instantaneous load state vector of each node in the heterogeneous resource pool, filtering the candidate node set, and selecting the node with the smallest current task concurrency as the allocation target.

[0117] After the atomic-level subtask to be migrated is reassigned to a new node, the computing power link orchestration module obtains the Euclidean distance between the resource utilization vector of the new node and the computational feature vector of the atomic-level subtask to be migrated. When the Euclidean distance is greater than a preset distance threshold, the SLA semantic guarantee module directly triggers model switching and routes the current inference request to the backup model execution engine. When the Euclidean distance is less than or equal to the preset distance threshold, the atomic-level subtask to be migrated is executed on the new node, and the output quality is collected again. If the semantic-level SLA commitment is still not met, the SLA semantic guarantee module triggers model switching.

[0118] Specifically, the output quality evaluator is deployed to the execution node along with each atomic subtask in the form of a lightweight container image. When an atomic subtask is assigned to a node, the scheduler first starts the output quality evaluator container on the node before starting the subtask execution process, and passes the subtask identifier, preset reference output sequence, and SLA threshold parameters to the evaluator. After the atomic subtask is completed, the execution process sends an evaluation trigger signal to the output quality evaluator. The signal carries the actual output sequence and task identifier of the subtask. Upon receiving the trigger signal, the output quality evaluator immediately starts the evaluation process.

[0119] The semantic similarity calculation submodule inputs the actual output sequence of the atomic-level subtask and the preset reference output sequence into the pre-trained semantic matching model. The semantic matching model is constructed and trained as follows: the Sentence-BERT architecture is used as the basic model, and the training data uses public datasets of natural language inference and semantic text similarity. Text pairs are input into the model, and the model encodes the two texts into fixed-dimensional semantic vectors. and Calculate the cosine similarity between two vectors. As the predicted similarity, the mean squared error loss function is used during training: ,in, The model training stops when the loss on the validation set no longer decreases, using the labeled true semantic similarity score. During actual inference, the semantic matching model outputs the semantic similarity score. The system uses the difference between 1 and the score as the inference accuracy loss value. The system employs a pre-defined reference output sequence. For requests with standard answers (such as knowledge-based question answering and mathematical calculations), the system retrieves the standard answer text from the knowledge base as a reference. For open-ended generation requests (such as summary generation), the system pre-executes the request using the same model during periods of ample resources, storing the output as a reference baseline in the cache. The pre-defined reference output sequence is obtained as follows: For reasoning requests with standard answers (such as closed-domain tasks like knowledge-based question answering, mathematical calculations, and code generation), the system retrieves the standard answer from the knowledge base as the pre-defined reference output sequence. For reasoning requests without standard answers (open-ended generation requests such as summary generation, creative writing, and open-domain dialogues), the system generates multiple output samples during reasoning by sampling multiple times (e.g., using the same model to generate multiple output samples for the same input under different temperature parameters (such as T=0.3, 0.7, 1.0)), using the average semantic similarity between the multiple output samples as the reasoning accuracy loss value.

[0120] The logical consistency detection submodule performs three independent logical checks on the actual output sequence: First, it checks for factual contradictions in the output. This is done by splitting the output text into propositional units, using a pre-trained natural language reasoning model (such as the RoBERTa-NLI model trained on the MNLI dataset) to determine if there are contradictory relationships between propositional units. If contradictory proposition pairs exist, the more contradictory proposition pairs there are and the higher the contradiction confidence, the lower the factual consistency score. The factual consistency score ranges from 0 to 1, with an initial value of 1. A preset deduction value (0.1 in the preferred embodiment) is deducted for each detected contradictory proposition pair until it reaches 0. Second, it checks for the completeness of the reasoning chain in the output, checking whether the causal and conditional relationships are complete. This is done by performing dependency parsing on the output text, extracting dependency arcs representing causal, conditional, and adversative logical relationships, and counting the actual number of existing logical relationship dependency arcs. Based on the logical reasoning depth value corresponding to this subtask Estimate the expected number of logical relationships. (Each reasoning chain contains an average of 2 logical relationship nodes), and the formula for calculating the completeness score of the reasoning chain is: The conclusion self-consistency score checks whether the final conclusion output is consistent with the intermediate conclusions in the reasoning process. Specifically, it extracts the set of intermediate conclusion clauses and the final conclusion clause from the output, encodes them as semantic vectors, calculates the average cosine similarity between the final conclusion vector and each intermediate conclusion vector, and uses this average as the conclusion self-consistency score. The system uses the weighted sum of the three scores as the logical coherence score. : ,in, Score for factual consistency. Score the completeness of the reasoning chain. Score the self-consistency of the conclusion. , and As the weighting coefficients, during the system initialization phase, several test samples labeled with quality scores for each dimension are selected. A grid search is then used to select the weight combination that yields the highest correlation between the weighted score and the human score. In this embodiment, The value is 0.3. The value is 0.4 and The value is 0.3; the system will calculate the logical coherence score. With logical coherence score threshold Compare and assign the inference accuracy loss value With the threshold of inference accuracy loss The comparison is as follows: when or When the current subtask output quality is deemed substandard, the SLA semantic guarantee module sends a migration instruction to the computing power link orchestration module. This migration instruction is sent in structured message format (e.g., JSON) and includes the following fields: Atomic-level subtask identifier to be migrated: a globally unique identifier composed of task ID and subtask sequence number; Current execution stage status identifier: including the batch processing round number already executed, the name and storage location of the generated intermediate result tensor, and the random number generator status sequence number; Type of substandard quality indicator: "Logical coherence substandard," "Accuracy loss exceeding limits," or "Both substandard"; and a snapshot of the source node's resource status: the resource occupancy vector and task status of the source node at the time of migration triggering. Upon receiving the migration instruction, the computing power link orchestration module parses the identifier of the atomic-level subtask to be migrated from the instruction. Starting from this identifier, the module performs a depth-first traversal in the dependency directed graph to find all downstream atomic-level subtasks that have this subtask as a prerequisite. For each downstream subtask found, the module sends a pause instruction to its execution node. The pause instruction contains the downstream subtask identifier and the pause reason code (indicating "precursor dependency task is being migrated"). After receiving the pause instruction, the execution node marks the execution status of the downstream subtask as "blocked and waiting", releases some of the computing resources it occupies (transfers intermediate results in GPU memory to CPU memory), and retains the minimum execution context for subsequent recovery.

[0121] After pausing downstream subtasks, the module unbinds the subtasks to be migrated from the currently assigned node. The unbinding operation includes: removing the subtask record from the current node's active task list; notifying the current node's execution engine to terminate any still running threads of the subtask; triggering a memory release operation on the current node; and marking the vertex state of the subtask in the dependent directed graph as "unassigned." Subsequently, the module reallocates the subtasks to be migrated to new nodes according to the aforementioned load balancing allocation method (filtering candidate nodes → selecting the node with the lowest concurrency → allocation). During reallocation, when calculating node resource utilization, the module converts the estimated size of the state snapshot into storage resource utilization, includes it in the node's resource utilization vector, and then performs the allocation.

[0122] After the subtask is reassigned to the new node, the module calculates the resource utilization vector of the new node. Computing feature vectors of subtasks Euclidean distance between ,in, to This represents the current occupancy rate of each type of resource on the new node. to The system will show the computational percentage of each subtask across the four dimensions. and preset distance threshold Comparison: A preset distance threshold is set by collecting data on the Euclidean distance of each migration task in the historical migration records and the success rate of execution after migration. With a post-migration execution success rate of no less than 80% as a constraint, the maximum corresponding Euclidean distance is selected as the threshold. ;when When the matching degree is insufficient, it indicates that the load distribution of the new node does not match the computational requirements of the subtask. At this time, even if it is re-executed, it is difficult to meet the SLA requirements. The SLA semantic guarantee module skips the re-execution step and directly triggers model switching, routing the current inference request to the backup model execution engine; when When the matching degree is sufficient, the module executes the subtask to be migrated on the new node. After execution, it calls the output quality evaluator again to collect the output quality. If the semantic-level SLA commitment is still not met ( or If the target model has fewer parameters, a model switch is triggered. The backup model is a model from the same series with fewer parameters than the target model. It can provide acceptable inference quality with lower resource consumption. The switch operation is as follows: the module forwards the input context of the current inference request and the results of the executed subtasks to the backup model execution engine. The backup model then re-executes the subsequent inference and returns the output of the backup model as the final result. The system records this switch event for subsequent system optimization analysis.

[0123] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A unified management and scheduling system for heterogeneous computing resources, characterized in that, Includes the following modules: Semantic parsing module: Used to receive reasoning requests and extract the cognitive complexity features of the requests in real time through a lightweight semantic router, including context length requirements, logical reasoning depth and result determinism requirements; Computing power link orchestration module: used to decompose the reasoning task into a sequence of sub-tasks with dependencies based on the cognitive complexity characteristics, and match the corresponding target computing power type for each sub-task. The computing power link orchestration module allocates each sub-task to the corresponding node in the heterogeneous resource pool through dynamic load balancing, and forms a global resource allocation scheme based on the dependencies. SLA Semantic Guarantee Module: Used to receive the cognitive complexity features to define a semantic-level SLA commitment, which includes a reasoning accuracy loss threshold and a logical coherence score threshold; During task execution, the SLA semantic guarantee module collects the actual output of each subtask and compares the logical coherence score and the inference accuracy loss value with the corresponding SLA thresholds. When any threshold is not met, the SLA semantic guarantee module sends a migration instruction to the computing power link orchestration module. After receiving the migration instruction, the computing power link orchestration module reallocates the subtasks to be migrated to the new node. The SLA semantic guarantee module collects the output quality again after the new node has completed the execution for judgment. During cross-node migration or model switching, the heterogeneous resource pool maintains the synchronization of task status through cache consistency overhead suppression mechanism and asynchronous communication mechanism, and reclaims the resources of abnormal nodes through memory leak detection and memory fragmentation suppression strategy.

2. The heterogeneous computing power resource unified management and scheduling system according to claim 1, characterized in that, The semantic parsing module extracts the cognitive complexity features of the request in real time through a lightweight semantic router, including: The inference request is input into a semantic encoder built on the Transformer encoder. The semantic encoder is deployed after end-to-end training for a cognitive complexity feature extraction task and outputs an encoded representation containing context semantic vectors and an attention weight matrix. The context length requirement is determined by statistically analyzing the distribution span of the number of times each input token is attended in the attention weight matrix, and then dividing the distribution span by the total number of input tokens. The inference request is matched with a preset set of logical connectors, and the ratio of the number of matches to the total number of input tokens is used as the logical connector density. The nesting level of each conditional clause in the inference request is analyzed, and the maximum nesting level is used as the recursion depth of the conditional clause. The weighted sum of the logical connector density and the recursion depth is used as the logical inference depth. Input the probability distribution of the output vocabulary corresponding to the reasoning request into the entropy calculation function to obtain the probability distribution entropy value; use the reciprocal of the probability distribution entropy value as the result certainty requirement.

3. The heterogeneous computing power resource unified management and scheduling system according to claim 1, characterized in that, In the computing power link orchestration module, the reasoning task is decomposed into a sequence of sub-tasks with dependencies based on the cognitive complexity characteristics, including: Obtain the structure configuration file of the target large model corresponding to the inference request. The structure configuration file contains several modal processing branches of the target large model, as well as the sub-modules and inter-layer connection relationships within each branch. Calculate the probability threshold based on the logical reasoning depth value in the cognitive complexity feature. ,in, The preset scaling factor is used. This represents the depth of logical reasoning. Along the forward propagation path when executing the inference request in the target large model, the inference task is divided into several modal-level subtasks with each modal processing branch of the target large model as the first-level segmentation boundary. For each modal-level subtask, the activation probability value of each sub-module contained in the modal-level subtask in the forward propagation path is traversed with the sub-module and inter-layer connection relationship as the second-level segmentation boundary. Sub-modules with activation probability values ​​< probability threshold are removed, and the remaining continuous sub-module sequence is taken as the atomic-level subtask of the inference task. Extracting the input tensor set of each atomic-level subtask With output tensor set For the first The atomic-level subtask and the first Atom-level subtasks, when and At that time, the judgment of the first The atomic-level subtask is the first The prerequisite tasks for each atomic-level subtask; when and At that time, the judgment of the first The atomic-level subtask and the first There are no dependencies between the atomic-level subtasks, and they can be executed in parallel. Traverse all pairs of atomic-level subtasks to generate a dependency directed graph with each atomic-level subtask as a vertex and the aforementioned prerequisite dependencies as directed edges.

4. The heterogeneous computing power resource unified management and scheduling system according to claim 3, characterized in that, The computing power link orchestration module matches the corresponding target computing power type for each subtask, including: Obtain the baseline performance vector of each node in the heterogeneous resource pool. The baseline performance vector includes the throughput per unit time of each node in four dimensions: floating-point operation, integer operation, matrix multiplication and addition operation, and logical branch operation. For any atomic-level subtask, extract the computational feature vector of the sub-module corresponding to the atomic-level subtask, which includes the computational proportion of the sub-module in the four dimensions; The calculated feature vector is multiplied by the baseline performance vector of each node to obtain the estimated execution efficiency score of the atomic subtask on each node. Using the result certainty requirement value in the cognitive complexity feature as a weighting coefficient, the estimated execution efficiency score on each node is multiplied by the weighting coefficient, and the computing power type corresponding to the node with the largest product is selected as the target computing power type of the atomic-level subtask.

5. A unified management and scheduling system for heterogeneous computing resources according to claim 4, characterized in that, The computing power link orchestration module allocates each subtask to the corresponding node in the heterogeneous resource pool through dynamic load balancing, including: The instantaneous load status vector of each node in the heterogeneous resource pool is collected in real time. The instantaneous load status vector includes the current occupancy rate of each type of resource and the current number of concurrent tasks of each node. For any atomic-level subtask with a selected target computing power type, nodes whose resource occupancy rate for each type is less than a preset occupancy threshold are selected from the heterogeneous resource pool to form a candidate node set. If the candidate node set is empty, calculate the L2 norm of the resource occupancy vector of each node and the cosine similarity between the resource occupancy vector and the calculated feature vector of the atomic subtask. Use the weighted sum of the negative of the L2 norm and the cosine similarity as the matching score of the node. Sort all nodes in descending order of matching score and add the top N nodes to the candidate node set until the candidate node set is not empty, where N is a preset expansion coefficient. From the candidate node set, the node with the smallest current task concurrency is selected as the allocation target; if there are multiple nodes with the same concurrency, the L2 norm of the resource utilization vector of each node is used as the load score, and the node with the smallest load score is selected as the allocation target.

6. The heterogeneous computing power resource unified management and scheduling system according to claim 3, characterized in that, In the computing power link orchestration module, a global resource allocation scheme is formed based on the dependencies, including: Using each atomic-level subtask as a vertex and the directed edges in the dependent directed graph as constraints, a task execution directed acyclic graph corresponding to the reasoning task is constructed. Perform critical path analysis on the directed acyclic graph for the task, and calculate the longest path length from any vertex to the exit vertex as the critical path length of the vertex. Based on the critical path length of each vertex in descending order, the vertices in the directed acyclic graph of the task are topologically sorted to generate an execution priority sequence for each atomic subtask. Based on the allocation results of each atomic-level subtask and the execution priority sequence, a global resource allocation scheme is generated, which includes the binding relationship between each atomic-level subtask and the target node, the execution priority of each atomic-level subtask, and the cross-node data transmission path.

7. The heterogeneous computing power resource unified management and scheduling system according to claim 1, characterized in that, In the SLA semantic guarantee module, a semantic-level SLA commitment is defined, including: comparing the feature values ​​of each dimension in the cognitive complexity feature with their respective preset thresholds, and determining the logical coherence score threshold and the reasoning accuracy loss threshold in the semantic-level SLA commitment based on the comparison results. The logical reasoning depth value is compared with a preset depth threshold. When the logical reasoning depth value is greater than the preset depth threshold, the logical coherence score threshold is set as a first coherence threshold; otherwise, it is set as a second coherence threshold. The result certainty requirement value is compared with a preset certainty threshold. When the result certainty requirement value is less than the preset certainty threshold, the reasoning accuracy loss threshold is set as a first loss threshold; otherwise, it is set as a second loss threshold. The context length requirement value is compared with a preset length threshold. When the context length requirement value is greater than the preset length threshold, the logical coherence score threshold is increased by a first correction amount, and the reasoning accuracy loss threshold is decreased by a second correction amount.

8. The heterogeneous computing power resource unified management and scheduling system according to claim 1, characterized in that, The SLA semantic guarantee module collects the actual output of each subtask and triggers cross-node migration or model switching, including: After each atomic-level subtask is completed, the output quality evaluator deployed in the execution node corresponding to the atomic-level subtask is invoked. The output quality evaluator includes a semantic similarity calculation submodule and a logical consistency detection submodule. The semantic similarity calculation submodule inputs the actual output sequence of the atomic-level subtask and the preset reference output sequence into the pre-trained semantic matching model to obtain the semantic similarity score output by the semantic matching model. The difference between 1 and the semantic similarity score is used as the reasoning accuracy loss value. The preset reference output sequence is obtained as follows: for reasoning requests with a standard answer, the standard answer is retrieved from the knowledge base as the preset reference output sequence; for reasoning requests without a standard answer, multiple output samples are generated through multiple samplings, and the average semantic similarity between the multiple output samples is used as the reasoning accuracy loss value. The logical consistency detection submodule performs factual consistency scoring, reasoning chain integrity scoring, and conclusion self-consistency scoring on the actual output sequence, and uses the weighted sum of the three scores as the logical coherence score. When the logical coherence score is detected to be less than the logical coherence score threshold or the inference accuracy loss value is greater than the inference accuracy loss threshold, the SLA semantic guarantee module sends a migration instruction to the computing power link orchestration module. The migration instruction carries the atomic-level subtask identifier to be migrated and the current execution stage status identifier. After receiving the migration instruction, the computing power link orchestration module queries all downstream atomic subtasks in the dependency directed graph that have the atomic subtask to be migrated as a prerequisite, suspends the execution of the downstream atomic subtasks, unbinds the atomic subtask to be migrated from the current allocation node, and reallocates the atomic subtask to be migrated to a new node by obtaining the instantaneous load state vector of each node in the heterogeneous resource pool, filtering the candidate node set, and selecting the node with the smallest current task concurrency as the allocation target. After the atomic-level subtask to be migrated is reassigned to a new node, the computing power link orchestration module obtains the Euclidean distance between the resource utilization vector of the new node and the computational feature vector of the atomic-level subtask to be migrated. When the Euclidean distance is greater than a preset distance threshold, the SLA semantic guarantee module directly triggers model switching and routes the current inference request to the backup model execution engine. When the Euclidean distance is less than or equal to the preset distance threshold, the atomic-level subtask to be migrated is executed on the new node, and the output quality is collected again. If the semantic-level SLA commitment is still not met, the SLA semantic guarantee module triggers model switching.

Images