A computing service orchestration method, electronic device and supernode server

Abstract

The application discloses a kind of computing service arrangement method, electronic equipment and supernode server, applied to computer technology field, to solve the problem that related technology cannot consider the adaptability of pre-training language model service, affect throughput and dynamic scheduling efficiency, difficult to adapt the high performance demand of LLM large-scale deployment.Proposed method includes: determining each to be scheduled task according to service job request;For each to be scheduled task, according to the task resource demand and task granularity scheduling demand of to be scheduled task, in combination with the node resource state information corresponding to each node in the cluster respectively, determine the target node corresponding to to be scheduled task;Through target node, run to be scheduled task, and the data generated in the process of to be scheduled task is cached to the corresponding cache node.It can better consider the service adaptability of pre-training language model, improve throughput and dynamic scheduling efficiency, so as to meet the high performance demand of pre-training language model large-scale deployment.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a computing service orchestration method, electronic device, and supernode server. Background Technology

[0002] Computing Power Network (CPN) is a new type of information infrastructure centered on computing, based on networking, and driven by intelligence. It can realize the local integration of computing, network, and storage resources, and provide reliable support for model services in intelligent computing scenarios of pre-trained language models (such as large language models, LLM) through unified orchestration and dynamic collaboration.

[0003] With the increasing heterogeneity of pre-trained language models and the continued volatility of workloads, CPN architectures based on static configuration and non-adaptive business processes in related technologies have limited adaptability and struggle to achieve intelligent, self-evolving optimization. For example, CPN architectures in related technologies primarily focus on resource-oriented or task-oriented single-dimensional orchestration optimization, which fails to consider the adaptability of pre-trained language model services. Furthermore, the low efficiency of computation and storage coupling affects throughput and dynamic scheduling efficiency, making it difficult to adapt to the high-performance requirements of large-scale LLM deployments. Summary of the Invention

[0004] This application provides a computing service orchestration method, electronic device, and supernode server to at least solve the problems in related technologies that cannot take into account the adaptability of pre-trained language model services, affect throughput and dynamic scheduling efficiency, and are difficult to adapt to the high-performance requirements of large-scale LLM deployment.

[0005] This application provides a computing service orchestration method, including: Determine each task to be scheduled based on the service job request; For each task to be scheduled, the target node corresponding to the task is determined based on the task resource requirements and task granularity scheduling requirements of the task to be scheduled, combined with the node resource status information of each node in the cluster. The task to be scheduled is run through the target node, and the data generated during the execution of the task to be scheduled is cached to the corresponding cache node.

[0006] This application also provides a computing service orchestration apparatus, including: The first determination module is used to determine each task to be scheduled based on the service job request; The second determination module is used to determine the target node corresponding to the task to be scheduled for each task based on the task resource requirements and task granularity scheduling requirements of the task to be scheduled, combined with the node resource status information of each node in the cluster. The execution module is used to run the scheduled tasks through the target node and cache the data generated during the execution of the scheduled tasks to the corresponding cache nodes.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement any of the above-described computing service orchestration methods.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described computing service orchestration methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described computing service orchestration methods.

[0010] This application also provides a supernode server, including: a memory and multiple computing nodes connected via an interconnection protocol; The memory stores a computer program, which, when executed by at least one computing node, enables the supernode server to implement the computing service orchestration method described above.

[0011] As can be seen from the above technical solution, the beneficial effects of this application are as follows: This application provides a computing service orchestration method, including: determining each task to be scheduled based on service job requests; for each task to be scheduled, determining a target node corresponding to the task to be scheduled based on the task resource requirements and task granularity scheduling requirements of the task to be scheduled, combined with the node resource status information corresponding to each node in the cluster; running the task to be scheduled through the target node, and caching the data generated during the running of the task to be scheduled to the corresponding cache node.

[0012] This application, upon receiving a service job request, identifies various tasks to be scheduled based on that request. For each task, it determines its resource requirements and granular scheduling requirements. Furthermore, by combining the node resource status information of each node in the cluster, it identifies the target node corresponding to each task. This allows a service job request to be run on multiple target nodes. Data generated during the execution of the tasks is cached on corresponding cache nodes. This application orchestrates computational services from multiple dimensions, including tasks and resources. By using target nodes for task computation and cache nodes for data storage, it not only better balances the service adaptability of pre-trained language models but also improves throughput and dynamic scheduling efficiency, thus meeting the high-performance requirements of large-scale deployment of pre-trained language models.

[0013] Furthermore, this application also provides corresponding electronic devices and supernode servers for the computing service orchestration method, further making the method more practical, and the electronic devices have corresponding advantages. Attached Figure Description

[0014] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A computing service orchestration architecture diagram provided for embodiments of this application; Figure 2 A flowchart illustrating a computing service orchestration method provided in this application embodiment; Figure 3 An architecture diagram of another computing service orchestration system provided in this application embodiment; Figure 4 A schematic diagram illustrating the execution process of another computing service orchestration method provided in this application embodiment; Figure 5 A comparison chart of training time costs in different scenarios provided for embodiments of this application; Figure 6 A comparison chart of startup time costs for a small batch dataset provided in this application embodiment; Figure 7 This is a structural diagram of a computing service orchestration device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0017] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0018] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] It's important to note that the surge in pre-trained language models has fundamentally altered the infrastructure supporting Artificial Intelligence (AI), driving a paradigm shift from model-centric computing to large-scale, networked intelligence. The increasing diversity of LLM-based service scenarios, such as large-scale inference, knowledge-based generation, and multimodal perception, not only requires real-time interactivity and deterministic latency guarantees but also ultra-high computational throughput across heterogeneous infrastructures. To meet these demands, a distributed environment capable of precisely coordinating computing, communication, and data resources is urgently needed to ensure scalability and reliability. In this context, Computing Networks (CPNs), as next-generation infrastructure, can achieve local integration of computing, network, and storage resources, providing reliable support for large-scale model services in large-scale intelligent computing scenarios through unified orchestration and dynamic collaboration.

[0020] In the CPN framework, a distributed scheduler coordinates multi-granularity resource allocation across heterogeneous computing nodes to ensure balanced utilization and minimize queuing latency under dynamic service demands. Beyond resource scheduling, CPN further extends coordination to the network layer, where deterministic transport mechanisms jointly optimize routing and flow control, maintaining bounded end-to-end latency and throughput stability under large-scale concurrency. Hierarchical data caching and location-aware placement reduce redundant transmission of model parameters, key-value (KV) caches, and intermediate data, thereby improving I / O (Input / Output) efficiency and service continuity. This meets the deterministic performance, scalability, and service continuity requirements of large model workloads across heterogeneous infrastructures.

[0021] With increasing model heterogeneity and workload volatility, CPNs based on static configuration and non-adaptive business processes in related technologies have limited adaptability and cannot support intelligent, self-evolving optimization. CPNs in related technologies suffer from three core problems: one-dimensional orchestration, inefficient computation-storage coupling, and coarse-grained responsive scheduling. Firstly, CPN orchestration in related technologies focuses primarily on one-dimensional, resource-oriented, or task-oriented optimization, ignoring cross-domain dependencies between data, models, and computational resources, thus limiting the adaptability and scalability of pre-trained language model services. Secondly, it suffers from decoupled deployment of computation and storage and insufficient data locality, leading to low caching efficiency and decreased throughput under large workloads. Thirdly, the reliance on coarse utilization metrics (e.g., CPU / GPU usage and memory consumption) and resource-driven schedulers lacking phased and structural awareness limits adaptive coordination and reduces efficiency under dynamic pre-trained language model workloads.

[0022] It should also be noted that the relevant technologies currently include the following: One approach is service orchestration and scheduling technology based on distributed caching. This typically uses heterogeneous node resources (such as compute, storage, memory, and CPU) as resource feature vectors, and extracts distributed cache features from both the model and data perspectives to construct model and data feature vectors. The model and data feature vectors are then evaluated using a pre-trained machine learning model, outputting an adaptive score for each node. Upon task arrival, the system selects the optimal node based on the score, forming a high-performance execution cluster to improve resource utilization and computational efficiency.

[0023] Another approach is hierarchical memory and interconnect optimization technology. This technology integrates access frequency and proximity characteristics with LRU (Least Recently Used) and LFU (Least Frequency Used) strategies to dynamically assess data temperature and enhance system I / O throughput through data prefetching by combining behavioral pattern analysis. At the communication acceleration layer, the system utilizes RDMA (remote direct memory access) and GPU passthrough technology to construct efficient CPU-GPU data transfer paths and low-latency, high-bandwidth full-to-full topology across GPU nodes, improving the performance of communication-intensive distributed training tasks.

[0024] RDMA allows one server to directly access the memory of another server without involving the operating system kernel, eliminating unnecessary data copying and context switching. The main advantages of RDMA are low latency and high throughput, making it particularly suitable for large-scale distributed computing and deep learning workloads with intensive data exchange. GPU passthrough refers to establishing a direct data path between the GPU and the network interface card (NIC) or storage device, enabling zero-copy data transfer between GPU memory and external devices. By bypassing the CPU and host memory, GPU passthrough significantly reduces transmission latency and overhead, improving the efficiency of inter-GPU collaboration, especially in large-scale parallel training tasks.

[0025] Another approach is intelligent data routing technology based on deep learning-based planning algorithms. In this technology, traffic routing is crucial for maintaining efficient data transmission. Addressing the limitations of traditional routing techniques in handling large-scale cluster traffic growth and dynamic service demands, this technology integrates AI algorithms into data routing optimization. First, HGNN (Hierarchical Graph Neural Network) is used to extract network topology features (e.g., bottom-layer modeling of link states, middle-layer analysis of path structure, and top-layer integration of global traffic distribution). Then, a graph converter serializes traffic demands, and an enhanced graph-global concern mechanism dynamically calculates the optimal path weights, thereby dynamically adjusting traffic paths for optimized allocation.

[0026] HGNN employs a multi-layered abstract architecture to achieve hierarchical representation learning of network topology. Through a bottom-up information propagation mechanism, it aggregates local link states, path relationships, and end-to-end traffic patterns to construct three feature spaces (link layer, path layer, and path layer). A cross-layer attention mechanism dynamically fuses features of different granularities, preserving underlying physical topology details while establishing high-level semantic connections. The graph converter combines the global attention mechanism of the transformer model with graph inductive bias, achieving sequential modeling of traffic demands through topological constraints in self-attention. By incorporating a resilient attention mechanism based on the graph adjacency matrix, network topology consistency is ensured.

[0027] The above-mentioned technologies all suffer from the following problems during implementation: they are mainly based on single-dimensional, resource-oriented, or task-oriented orchestration, ignoring the cross-domain interdependencies between data, models, and computing resources; they suffer from decoupled deployment of computing and storage and insufficient data locality; and they rely on coarse utilization metrics for responsiveness and resource-driven schedulers that lack stage-level and structural awareness.

[0028] In view of this, embodiments of this application provide a computing service orchestration method, which is based on, for example, Figure 1 The illustrated computing service orchestration architecture includes a model application layer, an AI scheduling layer, a data orchestration layer, an infrastructure layer, and a transport optimization layer. The model application layer is primarily responsible for deploying intelligent models as services and implementing model functionality in real-world business scenarios. Its scope covers several key AI domains, such as natural language understanding, computer vision, multimodal information fusion, and autonomous decision-making and control. In this application, the model application layer generates service job requests when executing model tasks based on user input and sends these requests to the AI ​​scheduling layer. The AI ​​scheduling layer determines the tasks to be scheduled and their requirements based on the service job requests. Then, by coordinating and scheduling all cluster resources, it selects the optimal target node for each task from the infrastructure layer and distributes the tasks to the corresponding target nodes for execution. Data generated during the execution of the corresponding tasks on the target nodes is cached by the data orchestration layer, which identifies the corresponding cache nodes from the infrastructure layer. All transport operations, including task distribution and data transmission, can be implemented based on the transport optimization layer.

[0029] In this application embodiment, the following is combined with Figure 2 The flowchart shown illustrates a computing service orchestration method, which is described in detail below. The method includes the following steps, S110 to S130.

[0030] S110: Determine each task to be scheduled based on the service job request.

[0031] It should be noted that when the model application layer executes model tasks, it generates a service job request and sends the service job request to the AI ​​scheduling layer. The AI ​​scheduling layer then breaks down the service job request into tasks, resulting in the corresponding tasks to be scheduled.

[0032] S120: For each task to be scheduled, based on the task resource requirements and task granularity scheduling requirements of the task to be scheduled, and combined with the node resource status information of each node in the cluster, determine the target node corresponding to the task to be scheduled.

[0033] In this embodiment, the task requirements corresponding to each task to be scheduled are obtained. These task requirements include task resource requirements and task granularity scheduling requirements. By monitoring the status of all nodes in the infrastructure layer, the node resource status information corresponding to each node in each cluster is obtained. Then, by analyzing the node resource status information of each node in each cluster, the corresponding target node is selected for each task to be scheduled.

[0034] S130: Run the scheduled task through the target node and cache the data generated during the execution of the scheduled task to the corresponding cache node.

[0035] For each task to be scheduled, the task is placed on the target node for task scheduling, that is, the corresponding task to be scheduled is run through the target node. During the execution of the task to be scheduled, various data will be generated. In this application, a cache node for caching data can be determined, and then the data is sent to the cache node for cache storage.

[0036] Please refer to Figure 3 , Figure 3 This application illustrates another computing service orchestration system provided in its embodiments. The system includes a model application layer, an artificial intelligence scheduling layer (AI scheduling layer), a transmission optimization layer, an infrastructure layer, and a data orchestration layer. The model application layer generates service job requests based on application scenarios, specifically including functional modules such as multimodal task aggregation and distribution, task classification and preprocessing, model processing, and decoding output. The AI ​​scheduling layer splits and orchestrates service job requests, and determines target nodes for tasks to be scheduled based on the resource status of each node. The infrastructure layer manages basic resources such as the graphics processing unit pool, central processing unit pool, and service cluster, and provides corresponding computing resources for tasks to be scheduled through resource decoupling. The transmission optimization layer performs data flow aggregation (including data flow classification and data flow merging), route topology awareness, and transmission node path planning between initial nodes and target nodes to improve data transmission efficiency between nodes. The data orchestration layer performs distributed storage, hybrid storage, and hierarchical data management to cache node data, distribute storage, and manage data in a hierarchical manner based on data access frequency. Therefore, computing services can be orchestrated in terms of tasks, computing resources, data storage, and data transmission, so that the tasks to be scheduled can run on the corresponding target nodes, and the data generated during the task execution can be cached to the corresponding cache nodes.

[0037] Please refer to Figure 3In this embodiment of the application, each node of each cluster is managed by the infrastructure layer. The infrastructure layer can decouple resources from each cluster, for example, by abstracting heterogeneous computing resources such as GPUs and CPUs into different resource pools, thereby achieving resource decoupling and physical independence. This approach can not only enable dynamic requesting and flexible allocation of resources for various computing tasks, but also improve the overall utilization efficiency of computing resources.

[0038] In this embodiment, the model application layer serves as the top-level module of the pre-trained language model AI native computing service cluster. It is responsible for deploying intelligent models as services and implementing model functionalities in real-world business scenarios. The core of the model application layer's implementation is the decoupling of model service-oriented architecture from business orchestration. At the lower level, trained models can be standardized and encapsulated using a unified inference framework and deployed to the computing cluster as containerized services, supporting on-demand scheduling, elastic scaling, and multi-version management. Through unified service interfaces (such as APIs (Application Programming Interfaces) or RPCs (Remote Procedure Calls), upper-layer businesses can call the model capabilities of the model application layer. Combined with dynamic batching, cache reuse, and resource-aware scheduling, low latency and high throughput are guaranteed in high-concurrency scenarios.

[0039] Furthermore, this application optimizes the data flow during data transmission through a transmission optimization layer. Since data transmission in a cluster setup typically generates a large number of concurrent small data streams, these streams may compete for the same network link resources, leading to network congestion and low transmission efficiency. Therefore, this application embodiment further reduces network congestion and bandwidth fragmentation by merging multiple small data streams into a smaller, larger data stream, thereby improving transmission efficiency and overall throughput.

[0040] In practical applications, the process of merging multiple small data streams into a smaller, larger data stream for transmission can include: The process involves acquiring the currently transmitting small data streams and grouping multiple small data streams with the same destination and origin into a small data stream group. These small data streams are then packaged and merged within the same group to obtain a corresponding large data stream, which is then transmitted to its destination. In other words, multiple small data streams with the same destination and origin are packaged into a larger data stream for transmission, or multiple data blocks are aggregated into a larger data packet according to traffic requirements for transmission. It should be noted that for... Figure 3A detailed description of the AI ​​scheduling layer and data orchestration layer shown below can be found in the following embodiments. Based on the above embodiments, the embodiments of this application combine... Figure 3 and Figure 4 The technical solution was further explained and optimized.

[0041] In one implementation, the process of determining the target node corresponding to the task to be scheduled in S120, based on the task resource requirements and task granularity scheduling requirements of the task to be scheduled, and in conjunction with the node resource status information corresponding to each node in the cluster, may include: Obtain the task resource requirements, computational granularity requirements, and mixed granularity requirements of the tasks to be scheduled; Based on the task resource requirements, computational granularity requirements, and mixed granularity requirements of the task to be scheduled, and combined with the node resource status information of each node in the cluster, a candidate node combination corresponding to the task to be scheduled is determined; the candidate node combination shall include at least one node. From the candidate node combinations, target nodes that meet the task resource requirements, computational granularity requirements, and mixed granularity requirements are identified.

[0042] It should be noted that in practical applications, service job requests carry characteristic information such as model type, model version, resource requirements (such as GPU memory, computing power, etc.), granularity requirements, latency requirements, and priority. Therefore, after determining each task to be scheduled based on the service job request, each task to be scheduled also has corresponding task characteristics (required resources, granularity requirements, granularity level, etc.). The granularity level can be data level, model level, or computation level. Based on the task characteristics of the task to be scheduled, the task resource requirements and task granularity scheduling requirements can be determined. The task granularity scheduling requirements include computation granularity requirements and mixed granularity requirements. Mixed granularity requirements include information such as data parallelism scale, model splitting method, and parameter synchronization frequency.

[0043] The AI ​​scheduling layer in this embodiment can continuously collect node resource status information of each node in each cluster through the scheduler. This node resource status information can include one or more combinations of node computing resources, node network status, node storage information, and data location information. It can also obtain task priorities and dependencies in the node's task queue. Node computing resources can include CPU / GPU type and available resources; node network status can include bandwidth, latency, and topology level; and node storage information and data location information. Resource and topology graphs can be modeled based on the node resource status information of each node, where nodes are vertices, network links are edges, and different levels (same machine, same rack, cross-cluster) are assigned different communication costs, thus providing real-time, structured input for subsequent decision-making. Then, combining the resource and topology graphs of each node, candidate node combinations for the task to be scheduled are determined. Furthermore, from these candidate node combinations, target nodes that meet the task resource requirements, computing granularity requirements, and mixed granularity requirements of the task to be scheduled are identified for running the task.

[0044] Considering that scheduling methods in related technologies neglect the imbalance of heterogeneous resources in GPU clusters, leading to resource fragmentation and low utilization, in one implementation, to ensure high system efficiency and stability under concurrent, heterogeneous, and dynamic workloads, the process of determining the candidate node combination corresponding to the task to be scheduled based on the task resource requirements, computational granularity requirements, and mixed granularity requirements of the task to be scheduled, combined with the node resource status information corresponding to each node in the cluster, can also be implemented in the following way.

[0045] Based on the node resource status information corresponding to each node and the physical and network topology corresponding to each node, a corresponding cost tree (such as GPU / host-rack-Pod-cluster) is constructed. At each node in the cost tree, the heterogeneous GPU resource status and network information are maintained in real time, such as GPU type and available quantity, bandwidth, latency and congestion status.

[0046] Based on the cost tree corresponding to each node, and combined with the task resource requirements, computational granularity requirements, and mixed granularity requirements of the task to be scheduled, the candidate node combination corresponding to the task to be scheduled is determined.

[0047] In practical applications, a heterogeneous GPU allocator and a resource-aware stage placer can be used to traverse the cost tree from bottom to top to evaluate the comprehensive cost of different subtrees carrying tasks or stages. This comprehensive cost can take into account communication overhead, topological distance, and resource fragmentation penalties, prioritizing GPU sets with low communication costs and high resource matching to determine candidate node combinations. By allocating resources in a topologically proximate manner and dynamically updating the cost evaluation, adaptive adjustment of resource allocation is achieved, thereby reducing cross-node communication latency and improving training speed and overall system throughput. In one implementation, the process of determining the target node from the candidate node combination that meets the task resource requirements, computational granularity requirements, and mixed granularity requirements may include: The comprehensive score of each node in the candidate node combination is determined according to the preset scoring function. Based on the comprehensive score of each node, target nodes that meet the requirements of task resources, computational granularity, and mixed granularity are determined.

[0048] It is understood that, in the embodiments of this application, after determining the candidate node combination corresponding to the task to be scheduled, each node in the candidate node combination can be comprehensively scored according to a preset scoring function to obtain the comprehensive score corresponding to each node. The lower the comprehensive score, the lower the overall execution cost. Therefore, the node with the lowest comprehensive score in the candidate node combination can be used as the target node of the task to be scheduled.

[0049] In this embodiment of the application, the preset scoring function can be established based on at least two of the following: matching degree, communication cost, data affinity level, task priority penalty parameter, and resource fragmentation impact parameter.

[0050] In practical applications, corresponding weights can be set for the calculation matching degree, communication cost, data affinity level, task priority penalty parameter, and resource fragmentation impact parameter. Then, based on the calculation matching degree, communication cost, data affinity level, task priority penalty parameter, and resource fragmentation impact parameter, and their respective weights, a preset scoring function can be constructed.

[0051] In this embodiment, the matching degree is determined based on the task resource requirements of the task to be scheduled, the task type, whether the node type matches, and whether the resources meet the requirements. The communication cost is determined based on the node topology and bandwidth constraints. The data affinity level is determined based on the storage location of the data required by the task to be scheduled and the task type. For example, for a GPU task, the required data is stored on the CPU, so the data affinity level is low. The task priority penalty parameter is determined based on the priority of the task to be scheduled. If the priority of the task to be scheduled is higher than the priority of each task currently being executed on the node, then a penalty parameter is added to each executing task, that is, these tasks are paused, and a penalty parameter is added to the score. The resource fragmentation impact parameter refers to whether the task to be scheduled can be placed entirely on the node, whether there is enough space and resources to accommodate it. If not, the task needs to be split into fragments and distributed to other nodes, then this impact parameter will be larger. In one embodiment, the process of determining the target node that meets the task resource requirements, computational granularity requirements, and mixed granularity requirements from the candidate node combination can also be implemented in the following way.

[0052] From the nodes of the candidate node combination, determine the first nodes that meet the task resource requirements, computational granularity requirements, and mixed granularity requirements. Obtain the task type of the task to be scheduled. If the task type is a high computational intensive task, determine the node with the strongest computing power from each first node as the target node. In the case of a data-intensive task, the node with the most free storage and free bandwidth resources among the first nodes is identified as the target node.

[0053] In other words, in this embodiment, each candidate node is first selected to meet the resource requirements, computational granularity requirements, and mixed granularity requirements of the task to be scheduled. Then, a target node is determined from the candidate node combinations based on the task type of the task to be scheduled. Specifically, for highly computationally intensive tasks, the first node with the strongest computational power can be selected as the target node; for data-intensive tasks, the first node with the most free storage and bandwidth resources is selected as the target node. This granular scheduling in this embodiment maximizes resource utilization and improves task parallelism.

[0054] In one implementation, determining the node with the strongest computing power from among the first nodes as the target node includes: For each first node, determine the number of floating-point operations per second for each first node; The node with the highest number of floating-point operations per second is identified as the node with the strongest computing power, and the node with the strongest computing power is selected as the target node.

[0055] Since the number of floating point operations per second (FLOPS) can reflect the computing power of a node, in order to determine the optimal target node in this embodiment, when the task type is a high computationally intensive task, the number of floating point operations per second of each first node can be calculated, and then the first node with the largest number of floating point operations per second can be used as the target node to run the corresponding scheduled task.

[0056] In practical applications, after the target node is determined, the tasks to be scheduled can be placed in the task execution queue or inference service entry of the target node. The target node can then pull or receive the tasks and execute them. Furthermore, the task execution results of the target node can be obtained.

[0057] It should be noted that during the process of running the scheduled task through the target node, the resource and network changes of the target node can be continuously monitored. When bandwidth fluctuations, node load changes, or the insertion of high-priority tasks are detected, re-scoring and local re-arrangement can be triggered (such as performing stage migration or resource reallocation).

[0058] In practical applications, the network status of the target node can be monitored during the execution of the scheduled task. If network congestion occurs at the target node, the optimal path corresponding to the scheduled task can be determined based on the task granularity scheduling requirements of the scheduled task and the node resource status information of each node, combined with a pre-trained deep graph convolutional neural network. Based on the optimal path, a new target node can be determined, and the scheduled task can then be migrated to the new target node for execution, thereby ensuring the normal and efficient operation of the task.

[0059] It should be noted that optimal path selection can be achieved by combining a transmission optimization layer with an adaptive traffic routing mechanism based on deep reinforcement learning (DRL). This DRL mechanism incorporates a deep graph convolutional neural network (DGCNN), which continuously learns and refines its knowledge of network topology, node, and link characteristics in real time, resulting in a pre-trained DGCNN for automated and intelligent optimization of routing paths. During the optimal path selection process, node resource status information and various link characteristics (such as bandwidth, latency, and packet loss rate) can be collected in real time. The task-level scheduling requirements of the task to be scheduled, the node resource status information of each node, and the various link characteristics are then input into the pre-trained DGCNN. DGCNN processes the graph structure of the network topology, encoding node and link features into embeddings, enabling DRL to understand the current network status and providing crucial topological information for traffic routing decisions. DRL learns from historical traffic data to progressively optimize routing strategies. Learning is based on a reward function, which guides the learning process by maximizing throughput and minimizing latency. Using network embedding information provided by DGCNN, the DRL agent makes decisions, selecting the optimal routing path or next hop, thus obtaining the optimal path. The routing selection method provided in this embodiment can better adapt to flowing traffic patterns and improve overall performance.

[0060] It should also be noted that, in order to improve processing efficiency, the data generated during the execution of the scheduled task is cached on the corresponding cache nodes, thereby separating computation and storage. For tasks that can be processed in parallel, this can improve the parallel processing efficiency. In this embodiment, the data cache nodes are determined through the data orchestration layer. In a large-scale native AI computing service cluster, the data orchestration layer is the core module responsible for data cache management and efficient access. Its main function is to provide high-throughput, low-latency, and scalable data access support for the upper-layer computing and scheduling modules, thereby ensuring stable data flow and efficient task execution in complex multi-task environments. In order to solve the transmission and storage needs of massive data, this embodiment integrates a distributed caching mechanism in the data orchestration layer. By dividing large-scale data into standardized data segments and distributing them evenly across multiple cache nodes, each cache node only needs to manage a specific range of data segments, thereby achieving load balancing and parallel access. This approach eliminates single-point bottlenecks, improves access efficiency under high concurrency, and supports horizontal scalability as needed.

[0061] In one implementation, the process of caching data generated during the execution of the task to be scheduled onto the corresponding cache node in S130 above may include: The data generated during the execution of the task to be scheduled is divided into multiple standardized data segments according to the preset segmentation rules. The cached data and metadata corresponding to each standardized data segment are cached on different cache nodes.

[0062] It is understood that in the embodiments of this application, the data generated by the target node during the execution of the task to be scheduled can be segmented by the data orchestration layer. The data can be segmented into multiple standardized data segments according to the preset segmentation rules. Then, a corresponding cache node is determined for each standardized data segment, and the cached data and metadata corresponding to the standardized data segment are cached on the corresponding cache node.

[0063] In practical applications, the massive amounts of data generated during task execution can be divided into multiple standardized data segments (chunks / segments) according to a preset fixed size or preset logical boundaries. Then, each standardized data segment is mapped to a different cache node for caching using consistent hashing or range sharding algorithms. Each cache node is only responsible for its assigned data segment range, managing the corresponding cached data and its metadata (such as version, validity period, and access count). When a subsequent read request arrives, a unified routing layer or client SDK (Software Development Kit, a toolkit provided for developers) can directly locate the target cache node storing each standardized data segment based on the preset sharding rules and the recorded information of the standardized data segments, thereby avoiding the single point of failure bottleneck caused by centralized access.

[0064] In practical applications, replication mechanisms and dynamic scaling can further enhance stability and throughput. Hot data can be automatically replicated across multiple nodes or its cache level can be increased, reducing pressure on individual nodes. When concurrency or data volume increases, the system only needs to redistribute some data segments after a new node is added to achieve horizontal scaling without affecting the overall service. Combined with asynchronous origin pull, failure eviction policies (such as LRU / LFU), and health monitoring mechanisms, this ensures high hit rates, low latency, and good load balancing capabilities in high-concurrency scenarios.

[0065] In one embodiment, the method may further include: For each normalized data segment in each cache node, the cache data type is determined based on the access status of the normalized data segment; When the cached data type is cold data, the standardized data segment is migrated from the memory layer to the disk layer.

[0066] It should be noted that for data cached for the first time, its corresponding standardized data segments can be cached in the memory layer of the respective cache node. For data that has already been cached, it is determined whether the cached data segments in the cache node are cold data or hot data. If they are cold data, they are moved to the disk layer; if they are hot data, they are moved to the memory layer.

[0067] In other words, the data orchestration layer in this application can also integrate a hierarchical memory management strategy co-designed by hardware and software to achieve intelligent data management based on access frequency. In practical applications, a frequency-based intelligent storage algorithm can be adopted, that is, frequently accessed data (i.e., hot data) is cached in high-speed memory, while infrequently accessed data (i.e., cold data) is automatically evicted and stored on the local disk layer. This approach not only alleviates memory resource pressure but also improves cache hit rate and overall access performance by dynamically adjusting the cache.

[0068] Understandably, in practical applications, we can first perform popularity awareness by maintaining lightweight access statistics (such as the number of hits in the last N seconds, the most recent access time, the read-write ratio, the average object size, etc.) for each cached data (or standardized data segment) to estimate the popularity score. The popularity score can represent the access situation. Then, the popularity score is updated in the metadata. In this application, the popularity score is not a simple number of times, but also has a decay feature (the newer the access, the higher the weight), so as to quickly keep up with the changes in hot topics.

[0069] Then, tiered placement or migration is performed. This means that storage media can be tiered according to performance (high-speed DRAM / persistent memory PMem / local SSD / HDD), and capacity levels and target hit rates can be set for each tier. During data writing or initial reading, data first enters the memory tier. A background thread continuously scans the heat score; standardized data segments with heat scores above the threshold remain in the memory tier or are promoted from other tiers to the memory tier. Standardized data segments with heat scores below the threshold are evicted to the disk tier according to an eviction policy (LRU / LFU / ARC or "lowest heat score first"). At this time, the index of the original memory tier location corresponding to the standardized data segment can be retained on the disk for quick retrieval. To avoid jitter (hot spots repeatedly going up and down), the hysteresis interval and minimum residence time can be increased. Furthermore, under increased memory pressure, the eviction intensity and cold data threshold can be increased; under decreased memory pressure, the eviction intensity and cold data threshold can be relaxed. In practical applications, hot data should be stored at higher levels (i.e., the memory layer) as much as possible, while cold data should be automatically stored at lower levels (disk layer or remote storage), and should be dynamically adapted to the load.

[0070] In addition, in this embodiment of the application, when it is necessary to read the required data segments during task execution, a job request can be generated. The data orchestration layer can also determine the data segment requirements based on the job request and the task execution graph. For example, it can parse the data identifiers explicitly declared in the job request (such as dataset version, time range, object ID), and then combine them with the input and output contracts defined by the task operators to map the logical data requirements to specific physical data chunks (shards) through indexes or metadata services, and determine the various data segments required by the task, generating a data dependency list. This allows the determination of the minimum set of data that each scheduled task needs to access during execution.

[0071] It should also be noted that, in terms of cluster management, this embodiment can pre-store data and models through a distributed caching mechanism to achieve cache sharing across multiple nodes. For example, model dependency files and shared modules can be packaged into an OCI (Open Container Initiative) standard image to achieve model and data standardization, cross-platform compatibility, and efficient distributed deployment, supporting parallel pulling between nodes and thus saving pulling time. Configuring an ImageVolume (i.e., an image container) for mounting facilitates sharing among multiple container Pods (multiple Pods referencing the same OCI standard image). Regarding task scheduling, adaptive allocation and global orchestration algorithms are applied to computational granularity and hybrid granularity (data, model) resources to select the optimal node configuration, thereby ensuring optimized resource allocation and efficient task execution in highly heterogeneous and job-intensive environments. Therefore, after receiving a service job request, each task to be scheduled can be determined based on that request. For each task, the resource requirements and granular scheduling requirements are determined. Then, by combining the node resource status information of each node in the cluster, the target node corresponding to each task is determined. Thus, a service job request is placed on multiple target nodes for execution. Furthermore, data generated during the execution of the tasks is cached on the corresponding cache nodes. This application orchestrates computational services from multiple dimensions of tasks and resources, performing task computation through target nodes and data storage through cache nodes. This not only better considers the service adaptability of pre-trained language models but also improves throughput and dynamic scheduling efficiency, thereby meeting the high-performance requirements of large-scale deployment of pre-trained language models.

[0072] Furthermore, this application achieves task allocation through cross-node collaboration. The key to multi-node collaboration lies in the reasonable division of computing tasks (i.e., the computing tasks corresponding to service job requests) to ensure efficient collaboration between nodes and reduce communication overhead. Based on factors such as network topology, bandwidth, and latency between nodes, the optimal node combination is selected for the task to avoid performance degradation due to network bottlenecks or excessive distance between nodes.

[0073] In a multi-node environment, load balancing is crucial for efficient resource utilization. This embodiment monitors the load status (such as CPU, memory, and disk I / O) and task progress of each node to automatically adjust task allocation, preventing excessive resource consumption or idle resources on certain nodes and achieving dynamic load balancing. Furthermore, a global resource awareness mechanism is used to perceive the real-time resource status of each node in the cluster, and considering task priority, node configuration, and topology, a global optimization algorithm selects the optimal node configuration. During task execution, the system dynamically adjusts based on real-time load, task status, and resource changes. In this embodiment, to optimize resources under high computational and storage loads, reinforcement learning algorithms can be used to optimize task scheduling strategies. This allows the scheduler to learn the optimal resource allocation method through trial and error. Through multiple experiments and model training, the system can gradually adapt to the characteristics of different types of tasks, ultimately providing the optimal resource scheduling solution.

[0074] Please refer to Figure 5 , Figure 5 The training performance of two deep learning models on three datasets is demonstrated. Model 1 is the first model, while Models 2 and 3 are the second models. That is, the first model is trained on dataset 1, and the second models are trained on datasets 2 and 3 respectively, forming three training combinations. Evaluations were conducted in three scenarios: cold start, cache warm-up, and distributed caching + cold start. In the cold start scenario, the deep learning job starts without any cached data, indicating that each data request triggers a remote I / O operation, leading to a data bottleneck. In the cache warm-up scenario, the job starts with a fully cached dataset, thus preventing cache loss during training. In the distributed caching + cold start scenario, the job also starts without cached data but utilizes the on-demand loading and training capabilities of the distributed caching system. The distributed caching effectively alleviates the data access bottleneck and reduces the total job completion time, even without pre-loaded data at startup. An exception was observed in the ResNet50-Cifar100 group: because the dataset is stored in '.pth' format and cannot be directly read, the job must wait for the dataset to be fully extracted before training can begin.

[0075] Please refer to Figure 6 , Figure 6This is a diagram illustrating the startup time cost of a small batch dataset. Figure 6 These are two Gantt charts consisting of 17 consecutive mini-batch training processes. The upper chart shows the time breakdown under the cold start method, and the lower chart shows the time breakdown under the distributed caching + cold start method. The prefetching mechanism of the distributed cache significantly reduces I / O latency by actively retrieving the required data samples, and further saves time by enabling synchronous data loading and training. Furthermore, the Gantt charts show that GPU idle time is significantly reduced after using the distributed caching system, indicating a significant improvement in GPU utilization. Therefore, the method provided in this application embodiment can greatly reduce the time overhead of the CPN architecture.

[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0077] Embodiments of this application also provide a computing service orchestration apparatus, please refer to... Figure 7 The schematic diagram shown illustrates the structure of a computing service orchestration apparatus, which includes: The first determining module 11 is used to determine each task to be scheduled based on the service job request; The second determining module 12 is used to determine the target node corresponding to the task to be scheduled for each task based on the task resource requirements and task granularity scheduling requirements of the task to be scheduled, combined with the node resource status information corresponding to each node in the cluster. The execution module 13 is used to run the scheduled task through the target node and cache the data generated during the execution of the scheduled task to the corresponding cache node.

[0078] In one embodiment, the second determining module 12 includes: The first acquisition unit is used to acquire the task resource requirements, computational granularity requirements, and mixed granularity requirements of the task to be scheduled. The first determining unit is used to determine the candidate node combination corresponding to the task to be scheduled based on the task resource requirements, computational granularity requirements and mixed granularity requirements of the task to be scheduled, combined with the node resource status information of each node in the cluster; the candidate node combination includes at least one node. The second determining unit is used to determine the target node from the candidate node combination that meets the task resource requirements, computational granularity requirements, and mixed granularity requirements.

[0079] In one embodiment, the second determining unit includes: The first determining subunit is used to determine the comprehensive score of each node in the candidate node combination according to the preset scoring function; The second determining subunit is used to determine the target node that meets the task resource requirements, computational granularity requirements, and mixed granularity requirements based on the comprehensive score of each node.

[0080] In one implementation, the preset scoring function is established based on the calculation of matching degree, communication cost, data affinity level, task priority penalty parameter, and resource fragmentation impact parameter.

[0081] In one implementation, the node resource status information includes one or more combinations of node computing resources, node network status, node storage information, and data location information.

[0082] In one embodiment, the second determining unit includes: The third determining subunit is used to determine each first node from the candidate node combination that meets the task resource requirements, computational granularity requirements and mixed granularity requirements. The fourth determining sub-unit is used to obtain the task type of the task to be scheduled. In the case of a high computational intensive task, the node with the strongest computing power is determined from each first node as the target node. The fifth determining sub-unit is used to determine the node with the most free storage and free bandwidth resources from the various first nodes as the target node when the task type is a data-intensive task.

[0083] In one implementation, the fourth determining subunit includes: The sixth determining subunit is used to determine the number of floating-point operations per second for each first node; The seventh determination sub-unit is used to determine the first node with the highest number of floating-point operations per second as the node with the strongest computing power, and to take the node with the strongest computing power as the target node.

[0084] In one implementation, the operating module 13 includes: The segmentation unit is used to segment the data generated during the execution of the task to be scheduled into multiple standardized data segments according to preset segmentation rules. The cache unit is used to cache the cache data and metadata corresponding to each standardized data segment to different cache nodes.

[0085] In one embodiment, the device may further include: The third determining unit is used to determine the cache data type for each standardized data segment in each cache node based on the access status of the standardized data segment; Migration units are used to migrate standardized data segments from the memory layer to the disk layer when the cached data type is cold data.

[0086] For a description of the features in the embodiment corresponding to the computing service orchestration device, please refer to the relevant description in the embodiment corresponding to the computing service orchestration method, which will not be repeated here.

[0087] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described computing service orchestration method embodiments.

[0088] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described computing service orchestration method embodiments at runtime.

[0089] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0090] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described computing service orchestration method embodiments.

[0091] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described computing service orchestration method embodiments.

[0092] Embodiments of this application also provide a supernode server, including: a memory and multiple computing nodes connected via an interconnection protocol; The memory stores a computer program, which, when executed by at least one computing node, enables the supernode server to implement any of the computing service orchestration methods described above.

[0093] It should be noted that the supernode server in this application embodiment refers to a computing device that is physically formed by multiple computing nodes connected through an efficient interconnection protocol, possesses unified memory addressing capabilities across multiple physical computing nodes, and logically appears as a single computing system to upper-layer software and users. Multiple computing nodes can share or uniformly access the memory resources in the supernode server, and at least one computing node can execute a computer program stored in its memory, enabling the supernode server to implement any of the aforementioned computing service orchestration methods.

[0094] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0095] The foregoing has provided a detailed description of a computing service orchestration method, electronic device, and supernode server provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A computing service orchestration method, characterized in that, include: Determine each task to be scheduled based on the service job request; For each task to be scheduled, the target node corresponding to the task is determined based on the task resource requirements and task granularity scheduling requirements of the task to be scheduled, combined with the node resource status information corresponding to each node in the cluster. The scheduled task is run through the target node, and the data generated during the execution of the scheduled task is cached to the corresponding cache node.

2. The computing service orchestration method according to claim 1, characterized in that, Based on the task resource requirements and task granularity scheduling requirements of the task to be scheduled, and combined with the node resource status information corresponding to each node in the cluster, the target node corresponding to the task to be scheduled is determined, including: Obtain the task resource requirements, computational granularity requirements, and mixed granularity requirements of the task to be scheduled; Based on the task resource requirements, computational granularity requirements, and mixed granularity requirements of the task to be scheduled, and combined with the node resource status information of each node in the cluster, a candidate node combination corresponding to the task to be scheduled is determined; the candidate node combination includes at least one node. From the candidate node combinations, target nodes that satisfy the task resource requirements, the computational granularity requirements, and the mixed granularity requirements are determined.

3. The computing service orchestration method according to claim 2, characterized in that, Determining target nodes from the candidate node combinations that satisfy the task resource requirements, the computational granularity requirements, and the mixed granularity requirements includes: The comprehensive score of each node in the candidate node combination is determined according to a preset scoring function; Based on the comprehensive score of each node, target nodes that meet the task resource requirements, computational granularity requirements, and mixed granularity requirements are determined.

4. The computing service orchestration method according to claim 3, characterized in that, The preset scoring function is established based on at least two of the following: matching degree, communication cost, data affinity level, task priority penalty parameter, and resource fragmentation impact parameter.

5. The computing service orchestration method according to claim 2, characterized in that, The node resource status information includes one or more combinations of node computing resources, node network status, node storage information, and data location information.

6. The computing service orchestration method according to claim 2, characterized in that, Determining target nodes from the candidate node combinations that satisfy the task resource requirements, the computational granularity requirements, and the mixed granularity requirements includes: From each node in the candidate node combination, determine each first node that satisfies the task resource requirements, the computational granularity requirements, and the mixed granularity requirements; Obtain the task type of the task to be scheduled. If the task type is a high computational intensive task, determine the node with the strongest computing power from each of the first nodes as the target node. When the task type is a data-intensive task, the node with the most free storage and free bandwidth resources among the first nodes is determined as the target node.

7. The computing service orchestration method according to claim 6, characterized in that, The node with the strongest computing power among the first nodes is determined as the target node, including: For each first node, determine the number of floating-point operations per second for each first node; The node with the highest number of floating-point operations per second is determined as the node with the strongest computing power, and the node with the strongest computing power is selected as the target node.

8. The computing service orchestration method according to any one of claims 1 to 7, characterized in that, Cache the data generated during the execution of the task to be scheduled to the corresponding cache node, including: The data generated during the execution of the task to be scheduled is divided into multiple standardized data segments according to a preset segmentation rule. The cached data and metadata corresponding to each of the standardized data segments are cached on different cache nodes.

9. The computing service orchestration method according to claim 8, characterized in that, Also includes: For each standardized data segment in each cache node, the cache data type is determined based on the access status of the standardized data segment; In the case where the cached data type is cold data, the standardized data segment is migrated from the memory layer to the disk layer.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the computing service orchestration method as described in any one of claims 1 to 9.

11. A supernode server, characterized in that, include: Memory and multiple compute nodes connected via interconnect protocols; The memory stores a computer program, which, when executed by at least one of the computing nodes, causes the supernode server to implement the computing service orchestration method as described in any one of claims 1 to 9.

Images