A data processing method, a computing node and a computing node cluster

Abstract

The application provides a data processing method, a computing node and a computing node cluster, which are used to improve the performance of the training stage and / or the inference stage of a large model. In the method: when the first computing node detects that a model needs to be deployed or the model needs to be expanded; the idle memory in the plurality of computing nodes included in the computing node cluster can be obtained to form a distributed memory node; then, through the distributed memory node, the computing node supports data reading and writing using memory semantics. In this way, by using the data reading and writing mode of memory semantics, the data access performance can be improved, the LLM training period can be shortened, the model loading time can be reduced, and storage resources can be saved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, a computing node, and a cluster of computing nodes. Background Technology

[0002] In the field of artificial intelligence (AI), large models such as large language models (LLMs) are attracting increasing attention due to their superior performance and wide range of applications. The size of the training data and the number of parameters directly affect the performance of LLMs. Furthermore, in some scenarios where models grow rapidly, challenges are posed to AI infrastructure, especially the high read and write performance requirements of traditional storage systems for high-performance data access.

[0003] For example, during the training phase of LLM, data access performance affects training throughput. The read / write efficiency of checkpoint files (a type of state file periodically archived for interruption recovery during training) also significantly impacts fault recovery in training tasks. Both of these factors contribute to an increase in the LLM training cycle time.

[0004] For example, during the inference phase of an LLM model, an excessively large LLM file can directly lead to increased model loading time, making it difficult to deploy flexibly.

[0005] In addition, for example, the Transformer architecture (a deep neural network model based on self-attention mechanism) requires memory to perform calculations during the inference phase, but the graphics card has limited video memory, while the main memory is largely idle, resulting in a waste of memory resources.

[0006] Therefore, improving the performance of large models during the training and / or inference phases is of great research significance. Summary of the Invention

[0007] This application provides a data processing method, computing nodes, and computing node clusters to improve the performance of large models during the training and / or inference phases, such as improving data access performance, shortening the LLM training cycle, reducing model loading time, and saving storage resources.

[0008] In a first aspect, embodiments of this application provide a data processing method applied to a first computing node. The method includes: detecting a preset event, the preset event being used to instruct the deployment of a model or the expansion of a model; responding to the preset event, acquiring free memory from M computing nodes to obtain a distributed memory node, the distributed memory node including M memory nodes originating from the M computing nodes, where M is a positive integer greater than 1; acquiring a target data shard from the M computing nodes, and preheating the target data shard data into the distributed memory node; and performing data reading and writing of model data according to the distributed memory node.

[0009] This method enables data access using memory semantics, resulting in greater bandwidth and improved data read / write speeds. Based on this, the loading speed of large models can be significantly increased, greatly enhancing the flexibility of large model inference deployment. Furthermore, large model deployment can be combined with cloud-native deployment, allowing for elastic deployment during the model inference phase. Each inference service can dynamically adjust the number of inference instances based on load changes, thereby reducing the cost of the large model inference phase.

[0010] In one possible implementation, preheating the target data shards into the distributed memory nodes includes: obtaining the address of the first target data shard from the second computing node; receiving a first physical address from the second computing node; and obtaining a first logical address of the first computing node based on the first physical address of the second computing node, wherein the first logical address is used to access the physical memory corresponding to the first physical address of the second computing node. Correspondingly, in another possible implementation, performing data read / write operations on the model data based on the distributed memory nodes includes: accessing the physical memory corresponding to the first physical address of the second computing node based on the first logical address.

[0011] In this implementation, by constructing distributed memory nodes, memory sharing among multiple memory nodes can be achieved, enabling data access based on memory semantics, resulting in greater bandwidth and improved data read / write speed.

[0012] In one possible implementation, preheating the target data shard into the distributed memory nodes includes: loading a second target data shard from persistent storage. In the above implementation, the method further includes: if the second target data shard is not found among the M computing nodes.

[0013] In this implementation, if a data shard cannot be obtained from the distributed memory node, it can still be obtained from persistent storage, thereby ensuring the accuracy of data processing.

[0014] In one possible implementation, the method further includes: receiving a scheduling request for processing a first dialogue request; and obtaining historical calculation results from the distributed memory nodes based on the dialogue identifier of the first dialogue request.

[0015] In this implementation method,

[0016] In one possible implementation, the model includes at least one of the following: a large language model, a multimodal model, and a large recommendation model.

[0017] In this implementation, distributed memory nodes are used to store computation results. This increases the scheduler's scheduling options, allowing it to select suitable compute nodes from the compute node cluster corresponding to the distributed memory nodes. This decouples the compute nodes and memory nodes, thereby achieving load balancing within the compute node cluster. The direct retrieval of cached objects based on memory semantics provided in this application embodiment can significantly improve processing performance and efficiency.

[0018] Secondly, this application provides a computing node including a processor and a memory; the memory is used to store computer program instructions; the processor is used to execute the computer program instructions stored in the memory, causing the computing node to perform the method described in the first aspect and its various possible designs. Exemplarily, the memory can be managed by memory nodes, and thus can be managed by distributed memory nodes.

[0019] Thirdly, this application provides a computing node cluster, which includes multiple computing nodes, such as the first computing node in the first aspect or any of the methods in the first aspect. Each computing node may include a memory node, and multiple memory nodes form a distributed memory node.

[0020] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a cluster of computing nodes, causes the cluster of computing nodes to perform the method described in the first aspect or any possible design of the first aspect.

[0021] Fifthly, this application provides a computer program product comprising a computer program or instructions that, when run on a cluster of computing nodes, cause the cluster of computing nodes to perform the method described in the first aspect or any possible design of the first aspect.

[0022] Sixthly, this application provides a chip system including a processor and a memory, wherein the memory stores instructions; when the instructions are executed by the processor, they implement the methods described in the first aspect or any possible design of the first aspect. The chip system may be composed of chips or may include chips and other discrete devices.

[0023] The beneficial effects of the second to sixth aspects mentioned above can be found in the beneficial effects of the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of an architecture for data access based on a distributed file system.

[0025] Figure 2A This is a schematic diagram of the system architecture for accelerating model loading provided in an embodiment of this application;

[0026] Figure 2B This is a schematic diagram illustrating the process of accelerating model loading provided in an embodiment of this application;

[0027] Figure 3 A schematic diagram of a system architecture applicable to a data processing method provided in this application embodiment;

[0028] Figure 4 This application provides a schematic diagram of a system architecture for a distributed memory node.

[0029] Figure 5 A schematic diagram of a data preheating system architecture provided in this application embodiment;

[0030] Figure 6 This application provides a schematic diagram of a data access system architecture.

[0031] Figure 7 This is a schematic diagram illustrating an application scenario of a data processing method provided in an embodiment of this application;

[0032] Figure 8 This is a schematic diagram illustrating another application scenario of a data processing method provided in an embodiment of this application;

[0033] Figure 9 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0034] Figure 10 This is a schematic diagram of the structure of a computing node cluster provided in an embodiment of this application. Detailed Implementation

[0035] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0036] In the description of the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0037] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0038] In one possible scenario, during the training and / or inference phases of large models, a distributed file system can be used for local file caching to accelerate data access. For example, Figure 1 This is a schematic diagram of an architecture for data access based on a distributed file system. A distributed file system can be understood as storing files in blocks across different nodes, thereby enabling parallel access and improving the efficiency and performance of data access.

[0039] like Figure 1 As shown, a distributed file system is built within the computing cluster, transforming the storage-computing separation into a unified storage-computing system. This allows a data warm-up process to be initiated during the training or inference phases of a large model, preloading data from persistent storage to the local distributed file system via a data orchestration system. Based on this, when an application needs to read a file, it can read the data through the user file system. The application's file reading can be triggered in response to the large model's training or inference phases.

[0040] In the above scenario, storage-compute separation refers to deploying data storage and data computation separately on different nodes or systems. In this architecture, memory nodes are responsible for data storage and management, while compute nodes are responsible for data processing and computation.

[0041] In-memory computing refers to deploying data storage and computation on the same node or system. In this architecture, data storage and computation share the same set of hardware resources, and data can be processed and computed directly on the compute nodes, reducing data transmission overhead.

[0042] Through such Figure 1 The distributed file system shown can improve data read performance to some extent, but the need for multiple data copies and state switching between kernel and user modes limits this performance improvement. Furthermore, as... Figure 1 The data orchestration process in the data orchestration system shown is relatively complex, requiring data preheating and orchestration in both time and space.

[0043] In view of this, embodiments of this application provide a data processing method, a computing node, and a computing node cluster. In this method, the idle memory of multiple computing nodes in the computing node cluster is used to form a distributed memory node; data access is performed through the distributed memory node. Therefore, this method can achieve data access through memory semantics, leveraging the high bandwidth characteristics of memory read / write to improve data read / write performance and accelerate data access caching in the AI ​​field.

[0044] The method provided in this application can be applied to AI high-performance computing scenarios and can serve as AI infrastructure. AI high-performance computing scenarios may include, but are not limited to, LLM scenarios, large recommendation model scenarios, and multimodal model scenarios. Specifically, it may include at least one of the following stages: training stage, inference stage, and fine-tuning stage for various models.

[0045] To facilitate understanding of the embodiments of this application, the following describes the terms or nouns that may be used in the embodiments of this application:

[0046] (1) AI Infrastructure: This refers to the comprehensive system that provides the necessary hardware, software, network, data, and other resources to support AI applications and services. It is the material foundation and technical guarantee for the development of AI technology, providing comprehensive support for the development, training, inference, and deployment of AI applications. The hardware component of AI infrastructure includes physical hardware resources such as servers, storage devices, and network equipment. The software component of AI infrastructure includes operating systems, database management systems, development tools, and various AI frameworks.

[0047] (2) AI Applications: These are applications that use artificial intelligence technology to enhance functionality, improve efficiency, or create entirely new experiences. AI applications can be applied to a wide variety of fields, including but not limited to: smart homes, healthcare, finance, education, and so on.

[0048] (3) Computing Nodes: These are devices with computing capabilities. At the hardware level, computing nodes are equipped with processors and memory. At the software level, applications can run on computing nodes. Computing nodes can serve as AI infrastructure, supporting the development, training, inference, and deployment of AI applications.

[0049] For example, a compute node can be a server, meaning a compute node corresponds to a physical machine. As another example, a server can also include multiple compute nodes; for instance, different compute nodes can correspond to different virtual machines or containers. It should be noted that the specific form of the compute node is not limited in the embodiments of this application.

[0050] For example, computing nodes can be combined with other computing nodes or memory nodes to form a cloud computing platform, constituting a computing node cluster. Optionally, a computing node cluster can correspond to one server or multiple servers.

[0051] (4) Memory nodes: used to store data in the cloud computing platform, and usually provide a variety of different storage methods, such as object storage, block storage and file storage.

[0052] Primarily used to provide storage space, memory nodes can also perform some computational processing. For example, a memory node can be a server, desktop computer, etc. In terms of hardware, memory nodes can have various hardware forms; this application embodiment does not limit the hardware form of the memory node.

[0053] In this embodiment of the application, each memory node may consist of free memory on each computing node.

[0054] Each of the multiple memory nodes includes a memory or storage medium, and each memory node can independently access data. Depending on the actual deployment method of the multiple memory nodes, different memory nodes can be connected through corresponding connection methods, and no specific limitations are imposed in this embodiment. For example, different memory nodes can be interconnected and perform data transmission (i.e., data movement or data interaction) through high-speed serial connection (compute express link, CXL), remote direct memory access (RDMA), or unified bus (UB), thereby realizing data transmission within the first computing node. The first computing node can improve the reliability and availability of data.

[0055] (5) Distributed memory nodes: These consist of memory nodes on multiple compute nodes. Each memory node manages the free memory on a compute node and acts as one of the distributed memory nodes. In essence, a distributed memory node includes free memory distributed across multiple compute nodes.

[0056] In addition, memory nodes in a distributed memory node can provide data read and write interfaces to AI applications on their respective computing nodes.

[0057] (6) Data preheating (or "data preheating"): This refers to the process of loading the required data into memory in advance before data querying. This process aims to optimize query response time, reduce cold start latency caused by data not being cached in memory, and thus improve the overall performance of the system and user experience.

[0058] (7) Data reading and writing (or “data access”): This can include data input (which can also be understood as “write operation” or “data storage”) and data output (which can also be understood as “read operation” or “data reading”).

[0059] The data reading and writing described in the embodiments of this application may include a data input process, a data output process, or both a data input process and a data output process. This application does not limit the actual data processing method.

[0060] The method provided in this application can accelerate model loading and provide high-speed data read and write capabilities. For example, Figure 2A This is a schematic diagram of a system architecture for accelerating model loading provided in an embodiment of this application. The system may include: an automated operation and maintenance system 210A, an operation and maintenance platform 210B, a model deployment system 220, a distributed memory management system 230, and a target computing node 240. The target computing node 240 may include a portion of a distributed memory node 241, an inference engine 242, and a data access software development kit (SDK) 243. It should be noted that the target computing node 240 includes one memory node of the distributed memory node 241, and the distributed memory node 242 includes memory nodes distributed across other computing nodes.

[0061] For example, such as Figure 2A The system shown can be deployed in a cluster of computer nodes, on different computing nodes, or on the same computing node. This application does not limit the specific deployment method.

[0062] based on Figure 2A The system architecture shown can achieve the following: Figure 2B The process shown is to accelerate model loading. For example, Figure 2B This is a schematic diagram illustrating the process of accelerating model loading as provided in an embodiment of this application. This process can be applied to a cluster of computing nodes and may include the following steps:

[0063] S201: A preset event was detected.

[0064] For example, the preset event could be that the automatic expansion of the automatic operation and maintenance system 210A triggers the model deployment system 220 to deploy the model.

[0065] Another example is that the preset event can also trigger the model deployment system 220 to deploy the model when the operation and maintenance engineer deploys / expands the system through the operation and maintenance platform 210B.

[0066] It's understandable that when scaling up is required, the model's capacity needs to be increased to support more users.

[0067] S202: In response to a preset event, the model deployment system 220 performs data preheating. For example, the model deployment system 220 instructs the distributed memory management system 230 to preload the required model data.

[0068] S203: The distributed memory management system 230 loads model data to the distributed memory node 241. It should be noted that the distributed memory node includes not only the memory nodes on the target computing node, but also the memory node interfaces on other computing nodes.

[0069] S204: Target compute node 240 quickly loads the model using memory semantics. Here, the target compute node can be understood as the compute node used to load the model.

[0070] For example, the target computing node 240 includes an inference engine 242 for processing the inference phase of a large model; a data access SDK 243 for reading and writing data through memory semantics, reading model data from the distributed memory node 241 to achieve fast model loading; the distributed memory node 243 can be used to store the loaded model data.

[0071] In this embodiment of the application, data reading and writing based on memory has the characteristic of high bandwidth. Therefore, by utilizing the idle memory on each computing node in the computing node cluster to form a distributed memory node, and performing data reading and writing through memory semantics, the speed and performance of data reading and writing can be improved, enabling fast model loading and shortening the time for model training and / or inference.

[0072] To facilitate understanding of the methods provided in the embodiments of this application, the following will provide a detailed description through multiple scenarios.

[0073] In one possible scenario, Figure 3 This is a schematic diagram of a system architecture applicable to a data processing method provided in an embodiment of this application. This system architecture may include, but is not limited to: an AI application 310, an operation and maintenance management system 320, and a large-scale model data acceleration system 330.

[0074] (1) AI application 310 can trigger, but is not limited to, the following application scenarios: training stage of large language model, inference stage of large language model, fine-tuning stage of large language model, training stage of multimodal model, inference stage of multimodal model, fine-tuning stage of multimodal model, training stage of recommendation large model, inference stage of recommendation large model, fine-tuning stage of recommendation large model.

[0075] (2) The operation and maintenance management system 320 may include resource monitoring and resource scheduling functions. Among them, resource monitoring can be used to detect scenarios that require expansion, and resource scheduling can be used to trigger expansion.

[0076] For example, the operation and maintenance management system 320 can provide, for example, Figure 2A The automated operation and maintenance system 210A shown can automatically perform expansion through resource scheduling when the resource monitoring function detects a need for expansion. Another example is the operation and maintenance management system 320, which can also be... Figure 2A The operation and maintenance platform 210B used by the operation and maintenance engineer shown can perform deployment or expansion processing through the resource scheduling function when the resource monitoring function detects that the operation and maintenance engineer manually triggers deployment / expansion.

[0077] (3) The large model data acceleration system 330 may include, but is not limited to: a data access SDK, distributed memory nodes, and a distributed memory control unit. For example, the large model data acceleration system 330 can be supported by a cluster of computing nodes.

[0078] (3.1) The data access SDK can provide memory-semantic-based data access APIs, offering different data type access interfaces for different scenarios. Furthermore, the data access SDK interacts with distributed memory nodes via shared memory, leveraging the high bandwidth of memory to achieve efficient data access. For example, the data access SDK may include, but is not limited to, the following SDKs: Tensor SDK (an SDK that provides tensor operations), KV-cache SDK (a technique for optimizing caching in large model inference processes), and embedding SDK (a technique applied to large models).

[0079] (3.2) Distributed memory nodes consist of memory nodes on each compute node. These memory nodes manage the free memory on the compute nodes, acting as a memory node within the distributed memory node and providing data access interfaces for AI applications on the distributed memory node. For example... Figure 3As shown, the distributed memory node may include memory node 1, memory node 2, and memory node 3. It should be noted that the number of memory nodes included in the distributed memory node in this embodiment is not limited and can be determined based on the number of computing nodes with free memory.

[0080] (3.3) The distributed memory control unit may include a storage control unit and a business control unit. The storage control unit coordinates and manages the data on each memory node and records the sharding information of the stored objects. The business control unit interfaces with the operation and maintenance management system to control relevant business logic such as data preheating and persistent dumping.

[0081] It is necessary to understand that, such as Figure 3 The system architecture described is merely an example. In practical applications, modules can be added or removed, or modules can be split or merged. This application does not impose specific limitations. The division of functional modules in the system architecture described above is illustrative and represents only a logical functional division. In actual implementation, there may be other division methods, and the specific functions performed by each functional module may also be divided in other ways. For example, the system architecture may be divided into more or fewer functional services based on other factors, or the functions of each service may be divided in other ways, or the system may not be divided into functional services but instead work as a whole. The names of the functional modules described above are merely examples, and the names of the functional modules in this application embodiment are not limited.

[0082] To facilitate understanding how to build Figure 3 Distributed memory nodes in Figure 4 This is a schematic diagram of another system architecture for a distributed memory node provided in an embodiment of this application.

[0083] The business control unit in the large model data acceleration system 330 obtains memory usage from the operation and maintenance management system 320. For example, the business control unit interacts with the operation and maintenance management system to obtain the memory usage of each computing node in the computing node cluster.

[0084] The business control unit in the large model data acceleration system 330 notifies each memory node to adjust memory.

[0085] After receiving a request from the business control unit to adjust memory, each memory node in the large model data acceleration system 330 loads and writes data back from persistent storage. For example, each memory node loads the corresponding data from persistent storage into memory, or evicts data from memory, and writes it back to persistent storage if the data has been modified.

[0086] The distributed memory nodes in the large model data acceleration system 330 notify the storage control unit to update metadata. This metadata update includes identifying newly loaded data and discarded data. For example, the storage control unit persistently stores the metadata in its own database to achieve the metadata update.

[0087] In this way, through the interaction between the distributed memory control unit included in the large model data acceleration system 330 and the operation and maintenance management system 320, it is possible to construct distributed memory nodes based on the idle memory of each computing node in the computing node cluster.

[0088] To facilitate understanding the data warm-up process, Figure 5 This is a schematic diagram of a data preheating system architecture provided in an embodiment of this application. Optionally, model deployment may include containerized deployment, virtual machine deployment, etc. Figure 5 This section uses containerized deployment as an example to illustrate the data preheating process, and demonstrates that the data preheating process and the containerized deployment model process can be parallel. Combined with... Figure 3 The operation and maintenance management system 320 may include a container scheduler for implementing resource scheduling functions. The container scheduler can be used to assist in the data preheating process and the containerized deployment model process.

[0089] (I) Data preheating process

[0090] During the data warm-up process, the business control unit in the large model data acceleration system 330 can interact with the operation and maintenance management system 320. For example, the business control unit may include a model warm-up control module.

[0091] The model preheating control module can be used to obtain the scheduling results from the container scheduler. These scheduling results can be interpreted as determining which compute nodes (or target compute nodes) the model will be deployed to when a new instance is deployed.

[0092] The model warm-up control module can also be used to notify the storage control unit to send data loading instructions to the memory nodes of the target compute node, such as memory node 1. For example, the model warm-up control module can be used to notify each memory node that needs to load data which memory nodes should be used to retrieve the currently missing data fragments. Figure 5 As shown, data can be obtained from memory nodes 2, 3, and 4. For example, memory node 1 on the target computing node can communicate with distributed memory nodes, including other memory nodes, to synchronize missing data shards to memory node 1. It can be understood that memory node 1 belongs to the target computing node currently undergoing data preheating processing, and therefore can also be understood as the target memory node.

[0093] (II) Containerized Deployment Model Process

[0094] During containerized deployment, kubelet (a container orchestration engine) responds to the container scheduler's scheduling, creating containers and then launching processes to load the model. During model loading, model data can be retrieved from distributed memory nodes. When AI applications access data via the data access SDK, they typically use shared memory to access data from multiple memory nodes within the distributed memory infrastructure. Therefore, data access via memory semantics offers higher read / write bandwidth compared to file interface access, thus accelerating data read / write efficiency.

[0095] In this embodiment, when the AI ​​application accesses data through the data access SDK, it first establishes a connection with the target memory node and performs logical address mapping between shared memory and the AI ​​application. To facilitate understanding of the data access process based on distributed memory nodes, Figure 6 This is a schematic diagram of a data access system architecture provided in an embodiment of this application.

[0096] AI application 310 obtains data fragment addresses from memory nodes through the data access SDK included in the large model data acceleration system 330. For example, the memory node could be... Figure 5 The memory nodes are 2, 3, or 4. In some possible embodiments, a data access priority can be set. Optionally, the priority can be determined based on the data fragments included in the memory node. For example, memory node 2 includes more data fragments than memory node 3, and memory node 3 includes more data fragments than memory node 4. Therefore, memory node 2 has a higher priority than memory node 3, and memory node 3 has a higher priority than memory node 4. That is, data fragments can be obtained from memory node 2 first.

[0097] The memory node segments the requested data and translates the logical address to the physical address. Then, the memory node returns the physical address of the data segment to the AI ​​application 310 via the data access SDK included in the large model data acceleration system 330.

[0098] Based on the interaction between the AI ​​application 310 and the memory node, the AI ​​application 310 can obtain the physical address corresponding to the data fragment in the memory node.

[0099] AI application 310 uses the data access SDK included in the large model data acceleration system 330 to perform CPU copying based on the physical address of the data fragment, thereby converting the physical address of the data fragment into a logical address that AI application 310 can recognize.

[0100] AI application 310 uses a large model data acceleration system 330, including a data access SDK, to access data based on logical addresses.

[0101] Based on the above Figure 6 The system architecture shown enables data access through memory semantics, thereby improving the speed of loading large models.

[0102] The solutions provided in the above embodiments of this application can be applied to data processing scenarios with large data volumes, such as various large model scenarios. Based on the foregoing... Figures 2A to 6 As described in the previous section, the following embodiments will further illustrate the data processing method provided by the embodiments of this application through two possible application scenarios.

[0103] Scenario A: Fast loading of large models

[0104] For example, Figure 7 This is a schematic diagram illustrating an application scenario of a data processing method provided in an embodiment of this application. Taking the rapid deployment of large models for inference services as an example, the model inference service can be triggered by AI applications. It may include the following steps:

[0105] For example, by means of Figure 3 The operation and maintenance management system 320 shown detects the model inference service and, in response, deploys the model. In one possible scenario, where the model is deployed using a containerized approach, the operation and maintenance management system 320 may include a container scheduler to implement resource scheduling functionality.

[0106] The following describes the data warm-up process performed by the compute node cluster:

[0107] First, the business control unit listens for scheduling results from the container scheduler. For example, the business control unit can listen to pods (a group of containers, which is the basic scheduling unit in Kubernetes) through the kubernetes API-server. When a pod is scheduled by the container scheduler, the business control unit will listen for pod update events, parse the pod metadata through the pod update events, and thus obtain information on which compute nodes the model to be deployed will be deployed to.

[0108] Then, the business control unit notifies the storage control unit to preheat the data based on the scheduling results. For example, the business control unit notifies the storage control unit which memory nodes in the distributed memory nodes need to preheat the data. Figure 7 As shown, the example uses memory node 1 on compute node 1 and memory node 2 on compute node 2 as examples of data that needs to be warmed up.

[0109] Correspondingly, the storage control unit performs preheating of missing data shards. For example, the storage control unit notifies the target memory nodes, including the target compute nodes, to obtain the missing data shards from the target memory nodes through distributed memory nodes. For instance, if compute node 1 needs to preheat the model, compute node 1 can be used as the target compute node, and memory node 1 on compute node 1 can be used as the target memory node. Data shards from memory node 2 can then be obtained through distributed memory nodes.

[0110] Optionally, memory node 1 can also load missing data from persistent storage. For example, if memory node 1 fails to load a certain data shard from any of the distributed memory nodes, it will load it from persistent storage.

[0111] Optionally, memory node 1 can retrieve data from memory node 2 or persistent storage via the Hypertext Transfer Protocol (HTTP) interface, and the specific interface method is not limited.

[0112] The following describes the deployment process of the compute node cluster execution model:

[0113] First, the container scheduler listens for scheduling results. Based on the scheduling results, a target compute node can be selected to deploy the container. For example, the target compute node could be compute node 1.

[0114] Then, the inference engine in compute node 1 reads model data from the distributed memory nodes via the data access SDK. For example, when the inference engine loads the model via the data access SDK, it can do so through methods such as... Figure 6 The method described is for copying model data.

[0115] Optionally, the data access SDK and memory nodes can communicate using domain sockets (a type of inter-process communication), and the specific communication method is not limited.

[0116] It should be noted that the data preheating process and model deployment process described above can be executed in parallel. This eliminates the need to rearrange and modify existing deployment processes, thus improving the efficiency of model deployment.

[0117] The method provided in this application significantly improves the loading speed of large models and greatly enhances the flexibility of large model inference deployment. Furthermore, large model deployment can be combined with cloud-native deployment, enabling elastic deployment during the model inference phase. Each inference service can dynamically adjust the number of inference instances based on load changes, thereby reducing the cost of the large model inference phase. Compared to using a distributed file system for data access, the method provided in this application utilizes memory semantics for data access, resulting in greater bandwidth and improved data read / write speeds.

[0118] Scenario B: LLM multi-turn dialogue scenario

[0119] User interactions with an LLM typically follow a question-and-answer format, involving multiple rounds of dialogue. Before initiating the current round of dialogue, the content of all previous rounds is inserted as prompts for the current LLM inference. Based on this, a store-based computation approach is adopted. Each inference step prioritizes querying the cache; if a cache hit occurs, the previous calculation result is retrieved directly; otherwise, the result is saved to the cache for use in the next round of dialogue. Compared to scenarios without store-based computation for multiple rounds of dialogue, this reduces the amount of repetitive inference of large amounts of dialogue content, saving computational resources.

[0120] In this embodiment, based on providing distributed memory nodes, previous computation results can be obtained by querying the distributed memory nodes. For example, when the first memory node is hit, the computation can be preferentially scheduled to the first computing node to which the first memory node belongs. It should be noted that this embodiment is not limited to scheduling to the first computing node; scheduling to the second computing node is also possible. The second computing node can also access the first memory node by accessing the distributed memory.

[0121] For example, Figure 8 This diagram illustrates another application scenario of a data processing method provided in this application. The method can be applied to a computing node cluster. The computing node cluster may include a scheduler. For example, the scheduler can be hosted on any computing node in the computing node cluster.

[0122] In one possible implementation, the scheduler schedules dialogue requests from the same session to the same compute node. For example, the scheduler queries the distributed memory nodes based on the dialogue identifier of the current dialogue request. When a matching computation result is detected, the scheduler obtains the compute node to which that result belongs and assigns the dialogue request to that compute node. This reduces cross-node data transfer within the distributed memory nodes.

[0123] In another possible implementation, the scheduler is used to schedule dialogue requests from the same session to any compute node in the same compute node cluster. For example, the scheduler queries the distributed memory nodes based on the dialogue identifier of the current dialogue request. When a matching computation result is detected, the scheduler obtains the compute node to which that result belongs and assigns the dialogue request to any compute node in the compute node cluster to which that compute node belongs. Optionally, a suitable compute node can be selected for processing based on its processing capacity, idle processing capacity, etc. It can be understood that any compute node in the compute node cluster can access data by accessing the distributed memory nodes.

[0124] For example, each computing node can process dialogue scenarios through the following implementation process:

[0125] First, during the AI ​​application startup phase, the data access SDK included in the compute node is initialized. For example, a namespace is created for the model through the memory node of the compute node, and the computation results of subsequent inference processes of the same model will be stored under this namespace.

[0126] Based on the computation results stored in the namespace, the scheduler can query whether the computation results belong to the same session as the dialogue request. If they exist, the target computation node is determined, and the dialogue request is scheduled to that target computation node.

[0127] Then, the inference engine in the LLM inference service included in the compute node calculates the block hash index. It should be noted that the inference engine in this scenario uses block-based memory management as an example.

[0128] The inference process in the inference engine creates cache objects. These cache objects are used to store cached data.

[0129] The data access SDK in the inference engine requests memory nodes to load historical key-value (kv) caches and load them into the memory of the created cache object.

[0130] The inference engine saves the current conversation in kvcache through the data access SDK.

[0131] The inference engine dereferences the kvcache object to free up memory space for the inference process.

[0132] The method provided in this application, which uses distributed memory nodes to store computation results, increases the scheduling options of the scheduler. It allows for the selection of suitable computing nodes from the computing node cluster corresponding to the distributed memory nodes, decoupling the computing nodes and memory nodes, thereby achieving load balancing within the computing node cluster. The method in this application, which directly retrieves cached objects based on memory semantics, can significantly improve processing performance and efficiency.

[0133] Reference Figure 9 To be applied Figure 2A or Figure 3 Taking the system architecture shown as an example, the flow of a possible data processing method provided in this application embodiment may include:

[0134] S901: The first computing node detects a preset event, which is used to indicate the deployment model or the expansion model.

[0135] S902: The first computing node responds to a preset event, obtains free memory from M computing nodes, and obtains a distributed memory node, wherein the distributed memory node includes M memory nodes originating from the M computing nodes, and M is a positive integer greater than 1.

[0136] S903: The first computing node obtains the target data shard from the M computing nodes and preheats the target data shard data into the distributed memory node.

[0137] S904: The first computing node performs data reading and writing of model data based on the distributed memory node.

[0138] It should be understood that the above implementation process is merely an illustrative example of the method process applicable to the embodiments of this application. The execution order of each step can be adjusted according to actual needs, and other steps can be added or some steps can be removed. The execution order between steps that are not temporally related can be arbitrary.

[0139] The specific steps performed by the first computing node in the above method can be found in the relevant descriptions in the foregoing embodiments, and will not be repeated here.

[0140] Based on the above embodiments and the same technical concept, this application also provides a computing node cluster for implementing the method provided in this application embodiment for a system composed of a first computing node and multiple other computing nodes. For example... Figure 10As shown, the computing node cluster includes one or more computing nodes 1000. Each computing node 1000 may include one or more processors 1001, memory nodes 1002, and one or more computer programs (not shown in the figure). In this embodiment, the memory nodes 1002 included in the multiple computing nodes 1000 can form a distributed memory node.

[0141] In one implementation, the aforementioned devices can be coupled via one or more communication lines 1003. The memory node 1002 stores one or more computer programs, each including instructions; the processor 1001 invokes the instructions stored in the memory node 1002, causing the computing node 1000 to execute the method provided in this embodiment for use on a first computing node.

[0142] In the embodiments of this application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0143] In this embodiment, the memory node can be a memory. The memory can be volatile memory or non-volatile memory, or a combination of both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. The memory in the embodiments of this application may also be circuitry or any other means capable of implementing storage functions.

[0144] In one implementation, the computing node 1000 may further include a communication interface 1004 for communicating with other devices via a transmission medium. For example, the computing node 1000 can interact with other nodes for information or data exchange through the communication interface 1004. In this embodiment, the communication interface may be a transceiver, circuit, bus, module, or other types of communication interface. In this embodiment, when the communication interface is a transceiver, the transceiver may include an independent receiver, an independent transmitter, or a transceiver with integrated transceiver functions, or an interface circuit.

[0145] In some embodiments of this application, the processor 1001, memory node 1002, and communication interface 1004 can be interconnected via a communication line 1003. The communication line 1003 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication line 1003 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0146] In some possible implementations, the memory nodes 1002 of one or more computing nodes 1000 in the computing node cluster may also each store a portion of the instructions for executing data processing methods. In other words, a combination of one or more computing nodes 1000 can jointly execute the instructions for executing data processing methods.

[0147] In some possible implementations, the connection method between different computing nodes 1000 in this computing node cluster can refer to... Figure 2A or Figure 3 The connection method between different memory nodes is shown.

[0148] The methods provided in this application can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., digital video disc (DVD), or semiconductor media (e.g., SSD), etc.

[0149] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, Applied to the first computing node, the method includes: A preset event is detected, which is used to indicate the deployment model or the scaling model; In response to a preset event, free memory from M computing nodes is obtained to obtain a distributed memory node, wherein the distributed memory node includes M memory nodes originating from the M computing nodes, and M is a positive integer greater than 1; Obtain target data shards from the M computing nodes and preheat the target data shards into the distributed memory nodes; Based on the distributed memory nodes, data reading and writing of model data are performed.

2. The method according to claim 1, characterized in that, The step of preheating the target data shards into the distributed memory nodes includes: Obtain the address of the first target data fragment from the second computing node; Receive the first physical address from the second computing node; Based on the first physical address of the second computing node, the first logical address of the first computing node is obtained, and the first logical address is used to access the physical memory corresponding to the first physical address of the second computing node.

3. The method according to claim 2, characterized in that, The process of reading and writing model data based on the distributed memory nodes includes: Based on the first logical address, access the physical memory corresponding to the first physical address of the second computing node.

4. The method according to any one of claims 1 to 3, characterized in that, Preheating the target data shards into the distributed memory nodes includes: Load the second target data fragment from persistent storage.

5. The method according to claim 4, characterized in that, The method further includes: The second target data shard was not found among the M computing nodes.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Receive a scheduling request, the scheduling request being used to process the first dialogue request; Based on the dialogue identifier of the first dialogue request, historical calculation results are obtained from the distributed memory nodes.

7. The method according to any one of claims 1 to 6, characterized in that, The model includes at least one of the following: a large language model, a multimodal model, and a large recommendation model.

8. A computing node, characterized in that, The computing node includes a processor and a memory; the memory is used to store computer program instructions. The processor is configured to execute computer program instructions stored in the memory, so that the computing node performs the method as described in any one of claims 1 to 7.

9. A computing node cluster, characterized in that, The computing node cluster includes multiple computing nodes, each of which includes a processor and a memory; the memory of at least one computing node is used to store computer program instructions. The processor of the at least one computing node is configured to execute computer program instructions stored in the memory of the at least one computing node, so that the cluster of computing nodes performs the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a cluster of computing nodes, causes the cluster of computing nodes to perform the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when run on a cluster of computing nodes, cause the cluster of computing nodes to perform the method as described in any one of claims 1 to 7.

Images