Data processing method, data processing device, and computer-readable storage medium
By reusing the actual model of the target node in Kubernetes, the problem of low utilization of graphics processor memory resources is solved, and efficient allocation and utilization of memory are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 青岛聚看云科技有限公司
- Filing Date
- 2026-01-19
- Publication Date
- 2026-06-05
Smart Images
Figure CN122152428A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of cloud-native technology, and in particular to a data processing method, a data processing device, and a computer-readable storage medium. Background Technology
[0002] The Kubernetes scheduler uses a request / limit model, linearly accumulating resource requests (Total Request = Σ(Pod Request)), leading to low node resource utilization. For example, if Pod A and Pod B use the same code or model, and both request 70GB of GPU memory from the Kubernetes scheduler, the scheduler assesses the remaining GPU memory on the nodes in the cluster. If it determines that only node A has remaining GPU memory (e.g., 80GB), the scheduler allocates 70GB of that memory to Pod A based on the order in which they requested it. If Pod A requested GPU memory before Pod B, and node A's remaining GPU memory (80GB) is greater than the GPU memory requested by Pod A (e.g., 70GB), the scheduler allocates it to Pod A. Then, because node A's current remaining GPU memory (10GB) is less than the GPU memory requested by Pod B (e.g., 70GB), the Kubernetes scheduler determines that resources are insufficient (70>10), and the Pod becomes BPending.
[0003] This results in the Kubernetes scheduler having to repeatedly allocate GPU memory to nodes when Pods with corresponding code or models execute within the same cycle, leading to low utilization of the node's GPU memory. Summary of the Invention
[0004] To address the aforementioned technical problems, this disclosure provides a data processing method, a data processing apparatus, and a computer-readable storage medium.
[0005] In a first aspect, this disclosure provides a data processing device, comprising: a communicator configured to: acquire creation information for creating a container group and runtime information of each worker node; wherein the creation information includes: a base model corresponding to the container group and dedicated graphics processing unit (GPU) memory for the container group, and the runtime information includes remaining GPU memory and one or more currently running actual models; and a controller configured to: determine the actual GPU memory corresponding to the base model; determine a target node based on the actual GPU memory, the base model, the dedicated GPU memory, and the runtime information, and bind the container group to the target node; wherein the target node includes any of the worker nodes; when an actual model identical to the base model exists, acquire the container identifier of the base model, and the environment variables and mounted volume of the actual model identical to the base model; and create a container group based on the container identifier, environment variables, mounted volume, creation information, dedicated GPU memory, and the actual model identical to the base model.
[0006] Secondly, this disclosure provides a data processing method, comprising: acquiring creation information for creating a container group and runtime information of each worker node; wherein the creation information includes: the base model corresponding to the container group and the dedicated graphics processor memory of the container group, and the runtime information includes the remaining graphics processor memory and one or more currently running actual models; determining the actual graphics processor memory corresponding to the base model; determining a target node based on the actual graphics processor memory, the base model, the dedicated graphics processor memory, and the runtime information, and binding the container group to the target node; wherein the target node includes any one of the worker nodes; when an actual model identical to the base model exists, acquiring the container identifier of the base model, and the environment variables and mounted volume of the actual model identical to the base model; and creating a container group based on the container identifier, environment variables, mounted volume, creation information, dedicated graphics processor memory, and the actual model identical to the base model.
[0007] Thirdly, this disclosure provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, the computer program being executed by a controller using a data processing method as provided in any of the second aspects.
[0008] Fourthly, this disclosure provides a computer program product that, when run on a computer, causes the computer to perform any of the data processing methods provided in the second aspect.
[0009] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be encapsulated together with the controller of the data processing device, or it may be encapsulated separately from the controller of the data processing device; this disclosure does not impose any limitations on this.
[0010] The descriptions of the second, third, and fourth aspects in this disclosure can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, and fourth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.
[0011] In this disclosure, the names of the aforementioned data processing devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this disclosure, they fall within the scope of this disclosure and its equivalents.
[0012] These or other aspects of this disclosure will become more readily apparent in the following description.
[0013] The technical solution provided in this disclosure has the following advantages compared with the prior art: The data processing device and communicator provided in this disclosure are configured to: acquire creation information for creating a container group and runtime information for each worker node; wherein the creation information includes: the base model corresponding to the container group and the dedicated graphics processor memory of the container group, and the runtime information includes the remaining graphics processor memory and one or more currently running actual models; the controller is configured to: determine the actual graphics processor memory corresponding to the base model; determine the target node based on the actual graphics processor memory, the base model, the dedicated graphics processor memory, and the runtime information, and bind the container group to the target node; wherein the target node includes any of the worker nodes; when an actual model identical to the base model exists, acquire the container identifier of the base model, and the environment variables and mounted volume of the actual model identical to the base model; and create the container group based on the container identifier, environment variables, mounted volume, creation information, dedicated graphics processor memory, and the actual model identical to the base model. In this way, when creating a new container group, if the target node has an actual model with the same base model as the container group, the data processing device can use the same actual model, container identifier, environment variables, mounted volumes, creation information, and dedicated graphics processor memory to create the container group. This achieves the reuse of the target node's graphics processor memory, improves the resource utilization of the graphics processor memory, and solves the problem in the existing technology where, when Pods with corresponding code or models execute in the same cycle, the Kubernetes scheduler needs to control the node to allocate memory repeatedly, resulting in low resource utilization of the node's graphics processor memory. Attached Figure Description
[0014] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 A schematic diagram of the data processing system provided in the embodiments of this application; Figure 2 One of the flowcharts illustrating the data processing method provided in the embodiments of this application; Figure 3 A second schematic flowchart illustrating the data processing method provided in this application embodiment; Figure 4 The third schematic flowchart of the data processing method provided in the embodiments of this application; Figure 5 The fourth flowchart illustrating the data processing method provided in this application embodiment; Figure 6 Fifth flowchart illustrating the data processing method provided in the embodiments of this application; Figure 7 A flowchart illustrating the data processing method provided in this application embodiment is shown in Figure 6. Figure 8 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application; Figure 9 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation
[0017] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0018] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0019] The data processing device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 2 and Figure 3This is one specific implementation of the data processing device of this application.
[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0021] In this disclosure, K8s refers to Kubernetes, which is an open-source container orchestration platform for managing containerized workloads and services.
[0022] In this embodiment of the disclosure, a Pod represents a container group containing one or more containers, which is the smallest and simplest deployment unit in Kubernetes.
[0023] In this embodiment of the disclosure, Deployment provides a declarative management approach for Pods and ReplicaSets.
[0024] CI / CD in this disclosure is a method for frequently delivering applications to customers by introducing automation during the application development phase, where CI stands for Continuous Integration and CD stands for Continuous Delivery or Continuous Deployment.
[0025] In this embodiment of the disclosure, kubectl is a command-line tool for Kubernetes, which acts like a "key" to interact with the Kubernetes cluster. Through it, you can send commands directly to the Kubernetes cluster from the terminal to manage various resources in the cluster, such as Pods, Deployments, and Services.
[0026] The Unix Domain Socket (UDS) request in this disclosure is primarily used to enable efficient and secure inter-process communication (IPC) between different processes on the same computer.
[0027] In this embodiment of the disclosure, KV Cache means key-value cache. It is a "memory warehouse" for the Transformer model during inference. It is specifically used to store previously calculated key and value matrices to avoid repeated calculations and make text generation faster and save memory.
[0028] In this embodiment of the disclosure, CRI stands for Container Runtime Interface, which is a container runtime interface defined in Kubernetes. It provides a standardized set of application programming interfaces (APIs) that enable Kubernetes to interact with various container runtimes (such as Docker, containerd, CRI-O, etc.).
[0029] The CRI Proxy in this disclosure is a Kubernetes tool for running multiple Container Runtime (CRI) implementations on a single node.
[0030] The OCI Runtime Hook (PreStart) in this embodiment is mainly used for container-level initialization operations, and resource injection (such as GPU driver) is achieved by modifying the container configuration.
[0031] In this embodiment, S3 stands for Amazon Simple Storage Service, which represents object storage service that manages data using buckets and objects, supporting massive storage and high reliability.
[0032] In this embodiment of the disclosure, OSS stands for Object Storage Service, which is compatible with S3API and provides various storage types such as standard, infrequent, and archive storage.
[0033] In some examples, the data processing methods provided in this disclosure are applied to, for example... Figure 1The data processing system shown includes a client 1, a scheduler 2, and a server cluster 3. The scheduler 2 includes a control plane 2-1 and a data plane 2-2. The server cluster 3 contains one or more servers, with each server corresponding to a worker node. The control plane 2-1 includes a target interface (such as an API Server) 2-1-1, a composite resource controller (Composite ReferenceController) 2-1-2, and an extended scheduler 2-1-3. The extended scheduler 2-1-3 includes a reference-aware scheduler 2-1-3-2. The data plane 2-2 includes a target node Kubelet 2-2-1, a Container Runtime Interface (Container Runtime CRI) 2-2-2, a resource proxy agent 2-2-3, a device layer 2-2-4, an artifact manager daemon 2-2-5, and a physical resource pool (including graphics processing units (GPUs), memory, etc.) 2-2-6.
[0034] In the following embodiments, the scheduler 2 is used as the execution subject of the data processing method provided in the embodiments of this disclosure to illustrate the method of the embodiments of this application.
[0035] This application provides a data processing method, such as... Figure 2 As shown, the data processing method may include S11-S15.
[0036] S11. Obtain the creation information for creating the container group and the running information of each worker node; wherein, the creation information includes: the basic model corresponding to the container group and the dedicated graphics processor memory of the container group, and the running information includes the remaining graphics processor memory and one or more currently running actual models.
[0037] In some examples, when a user needs to deploy a Pod in server cluster 3, the user can perform the deployment operation on client 1, such as submitting a Deployment on client 1. For instance, client 1 can be kubectl or CI / CD, in which case the user can submit YAML (i.e., Deployment) via kubectl or CI / CD. The user doesn't need to care about the specific physical memory address; they only need to declare two types of requirements: the base model corresponding to the Pod, such as a shared reference (e.g., cloud-native.jucloud / model-base: llama-3-70b); and the Pod's dedicated graphics processor memory, such as an exclusive request (e.g., nvidia.com / gpu-mem: 2GB, used to store embedded parameters, the adapter, and the KV cache). Afterward, client 1 responds to the user's deployment operation by sending creation information to scheduler 2. When scheduler 2's API Server 2-1-1 receives the creation information, it obtains the running information of each worker node through extended scheduler 2-1-3. Next, API Server 2-1-1 intercepts the creation information, for example, using Kubernetes MutatingAdmissionWebhook to intercept it, so as not to execute the creation information initially (this is because it's necessary to determine if a shared model exists, thus reducing the GPU memory usage of the same model). The Composite Resource Scheduler 2-1-2-1 queries the backend metadata and learns that the basic model llama-3-70b requires dedicated GPU memory, such as 140GB of dedicated frame buffer. Then, the Composite Resource Scheduler 2-1-2-1 injects a custom virtual resource type (e.g., virtual / shared-memory:140GB) into the Pod's resources.requests. This is not only a marker, but also to make the scheduler aware of this requirement. Next, the composite resource scheduler 2-1-2-1 sends the modified Pod configuration information to API Server 2-1-1, including: the base model corresponding to the container group, the actual graphics processor memory corresponding to the base model (such as 140GB of dedicated frame buffer), and the dedicated graphics processor memory of the container group. API Server 2-1-1 saves the modified Pod configuration information. At this point, the Pod appears to be requesting a huge amount of resources, but this lays the groundwork for the subsequent "exemption".
[0038] Next, API Server 2-1-1 sends the Pending Pod to Reference Aware Scheduler 2-1-3-2. Then, Extended Scheduler 2-1-3 determines the target node based on the actual GPU memory Shared_Size, the base model, the dedicated GPU memory Exclusive_Size, and runtime information, and binds the container group to the target node. This differs from the native scheduler, which only considers Node.Status.Allocatable (remaining amount). The Extended Scheduler 2-1-3 in the data processing method provided in this embodiment maintains a Node Content View, recording the remaining GPU memory of each worker node and the model identifier of the currently running model in each worker node, such as the Shared Artifact ID. Then, the extended scheduler 2-1-3 determines the target node based on the actual GPU memory, the base model, the dedicated GPU memory, and the running information. For example, for a worker node that does not have an actual model identical to the base model, one or more first nodes are determined based on the actual GPU memory, the dedicated GPU memory, and the remaining GPU memory. For example, in scenario A (worker node without cache): theoretical GPU memory Cost = Shared_Size (140GB) + Exclusive_Size (2GB) = 142GB. Result: If the worker node has 80GB of GPU memory remaining, the scheduling fails because 80GB is less than 142GB. For a worker node that has an actual model identical to the base model, one or more second nodes are determined based on the dedicated GPU memory and the remaining GPU memory. For example, in scenario B (worker node with cache): Cost = Shared_Size (0) + Exclusive_Size (2GB) = 2GB. Result: Even though the worker node has 80GB of GPU memory remaining, although 80GB is less than 142GB, the scheduling is successful because the worker node is running the same actual model as the base model. Since the actual model can be shared to create container groups, there is no need to create the same base model on the worker node. After that, the first score of the first node and the second score of the second node are obtained; based on the first and second scores, the target node is determined. The technical significance of determining the target node is as follows: Shared_Size (0) means that the base model shares the same actual model as the base model, so there is no need to create the base model again. Therefore, there is no additional GPU memory usage, so Shared_Size is 0. In this way, logical resource overselling is achieved, but it is physically safe (because physically only 2GB of Exclusive_Size is needed).
[0039] Next, the extended scheduler 2-1-3 obtains the operational information of each theoretical node. The theoretical node includes either the first node or the second node. The operational information includes one or more of the following: CPU utilization, instruction cache, data cache, and network parameters: speed, bandwidth, throughput, latency, utilization, and packet loss rate. Based on this operational information, a theoretical score is determined for each theoretical node; the theoretical score includes either the first score or the second score. Then, the extended scheduler 2-1-3 binds the Pod to the node with the "lowest cost" (i.e., the target node) and sends the binding information to API Server 2-1-1. API Server 2-1-1 then issues the Pod task to the target node. Upon receiving the Pod task from API Server 2-1-1, the target node sends information to the runtime injection proxy (CR Proxy / OCI Hook) to call CRI to create a container. When the runtime injection agent receives a request to create a container using CRI, it intercepts the container startup request (e.g., the agent could be an OCI (Open Container Initiative) Runtime Hook (PreStart) or a CRI Proxy). The runtime injection agent parses the Pod's configuration information (annotations) and finds that it requires shared-obj-v1. The runtime injection agent then initiates a Unix Domain Socket request to the shared artifact daemon on the target node. The shared artifact daemon then checks the GPU's video memory (GPU / Accelerator). If the shared artifact daemon determines that no actual model with the same base model is running on the target node, it downloads the model from object storage (S3 / OSS), decompresses it, and allocates cudaMalloc to the GPU's video memory, locking the memory segment and not releasing it. Alternatively, if the shared artifact daemon is running an actual model with the same base model on the target node, it directly obtains the IPC Handle of the memory segment where the existing actual model is running (e.g., a handle generated by CUDA's cudaIpcGetMemHandle). Simultaneously, the Daemon increments the reference count for the resource. Afterward, the shared artifact manager returns a cross-process handle (e.g., PCHandle / szhm_id) to the runtime injection agent. The runtime injection agent modifies the configuration information for the Pod task about to start (e.g., environment variables and mounted volumes of the actual model identical to the base model), and / or performs device mounts, such as potentially mounting / dev / shm or specific device files.Afterwards, the runtime injection agent sends information to the target node containing the container identifier of the underlying model to continue the startup process. From the target node's perspective, it only knows that the container has started normally. Then, the target node starts the business logic code within the Pod (such as a modified PyTorch or vLLM backend). The code reads the environment variable SHARED_MEMORY_HANDLE and calls cudaIpcOpenMemHandle. This allows 140GB of model weight data to appear in the process's virtual address space instantaneously (on the order of microseconds).
[0040] In some examples, scheduler 2 can be a Kubernetes scheduler.
[0041] S12. Determine the actual graphics processor memory corresponding to the basic model.
[0042] In some examples, the composite resource scheduler 2-1-2 determines the actual graphics processor memory corresponding to the base model by querying the background metadata. For example, when the base model is llama-3-70b, the composite resource scheduler 2-1-2 learns from the background metadata that the base model llama-3-70b requires 140GB of graphics processor memory. Therefore, the actual graphics processor memory corresponding to the base model llama-3-70b is 140GB.
[0043] S13. Based on the actual graphics processor memory, basic model, dedicated graphics processor memory, and runtime information, determine the target node and bind the container group to the target node. The target node includes any one of the worker nodes.
[0044] In some examples, the composite resource scheduler 2-1-2 can input the actual graphics processor memory, base model, dedicated graphics processor memory, and runtime information into the filtering model for filtering to determine the target node; wherein, one filtering model corresponds to one server cluster 3, and the training process of the filtering model includes: Obtain the first training sample data and the first labeling result of the first training sample data; wherein, the first training sample data includes at least one set of filtering information in the target server cluster, the first labeling result includes the filtering nodes corresponding to each set of filtering information in the target server cluster, one set of filtering information corresponds to one filtering node, the filtering node is any one of the working nodes contained in the target server cluster, and the filtering information includes the actual graphics processor memory, the basic model, the dedicated graphics processor memory, and the running information.
[0045] The first training sample data is input into the first neural network model for learning, and the first prediction result of the first neural network model on the first training sample data is obtained.
[0046] Based on the first prediction result and the first labeling result, the network parameters of the first God General Network model are adjusted until the first neural network model converges, thus obtaining the selected model.
[0047] In some examples, the composite resource scheduler 2-1-2 determines target nodes based on actual graphics processor memory, the base model, dedicated graphics processor memory, and runtime information. This includes: for working nodes that do not have an actual model identical to the base model, determining one or more first nodes based on actual graphics processor memory, dedicated graphics processor memory, and remaining graphics processor memory; for working nodes that have an actual model identical to the base model, determining one or more second nodes based on dedicated graphics processor memory and remaining graphics processor memory; obtaining a first score for the first node and a second score for the second node; and determining the target node based on the first and second scores.
[0048] S14. If an actual model identical to the base model exists, obtain the container identifier of the base model, as well as the environment variables and mounted volume of the actual model identical to the base model.
[0049] S15. Create a container group based on container identifier, environment variables, mounted volume, creation information, dedicated graphics processor memory, and the same actual model as the base model.
[0050] As described above, the data processing method provided in this embodiment obtains creation information for creating a container group and runtime information of each working node; determines the actual graphics processing unit (GPU) memory corresponding to the base model; determines the target node based on the actual GPU memory, base model, dedicated GPU memory, and runtime information, and binds the container group to the target node; wherein the target node includes any one of the working nodes; when an actual model identical to the base model exists, obtains the container identifier of the base model, as well as the environment variables and mounted volume of the actual model identical to the base model; and creates a container group based on the container identifier, environment variables, mounted volume, creation information, dedicated GPU memory, and the actual model identical to the base model. Thus, when creating a new container group, if an actual model identical to the base model corresponding to the container group exists in the target node, the device can use the actual model, container identifier, environment variables, mounted volume, creation information, and dedicated GPU memory corresponding to the base model corresponding to the container group to create the container group, thereby achieving reuse of the GPU memory of the target node and improving the resource utilization rate of the GPU memory.
[0051] In some feasible examples, combining Figure 2 ,like Figure 3 As shown, the above S13 can be specifically implemented through the following S130-S133.
[0052] S130. For working nodes of actual models that do not have the same model as the base model, determine one or more first nodes based on the actual graphics processor memory, dedicated graphics processor memory, and remaining graphics processor memory.
[0053] In some examples, the composite resource scheduler 2-1-2, for work nodes that do not have an actual model identical to the base model, determines one or more first nodes based on the actual GPU memory, dedicated GPU memory, and remaining GPU memory, including: Based on the actual graphics processor memory and the dedicated graphics processor memory, the theoretical graphics processor memory is determined; when the remaining graphics processor memory is greater than the theoretical graphics processor memory, the working node with the remaining graphics processor memory greater than the theoretical graphics processor memory is taken as the first node.
[0054] S131. For working nodes that have the same actual model as the base model, determine one or more second nodes based on the dedicated graphics processor memory and the remaining graphics processor memory.
[0055] In some examples, the composite resource scheduler 2-1-2, for working nodes with the same actual model as the base model, determines one or more second nodes based on dedicated graphics processor memory and remaining graphics processor memory, including: When the remaining GPU memory is greater than the dedicated GPU memory, the working node with the greater remaining GPU memory is designated as the second node.
[0056] S132, Get the first score of the first node and the second score of the second node.
[0057] In some examples, the composite resource scheduler 2-1-2 obtains the first score of the first node and the second score of the second node, including: obtaining the operating information of each theoretical node; wherein, the theoretical node includes any one of the first node and the second node, and the operating information includes one or more of the following: CPU utilization, instruction cache, data cache, and network parameters, and the network parameters include one or more of the following: rate, bandwidth, throughput, latency, utilization, and packet loss rate; and based on the operating information, determining the theoretical score of each theoretical node; wherein, the theoretical score includes any one of the first score and the second score.
[0058] In some examples, the composite resource scheduler 2-1-2 determines the theoretical score of each theoretical node based on runtime information, including: The operational information of the theoretical nodes is input into the scoring model for calculation to determine the theoretical score corresponding to each theoretical node. The training process of the scoring model includes: Obtain the second training sample data and the second labeling result of the second training sample data; wherein the second training sample data includes the running information of at least one working node, and the second labeling result includes the actual score corresponding to each working node.
[0059] The second training sample data is input into the second neural network model for learning, and the second prediction result of the second neural network model is obtained.
[0060] Based on the second prediction result and the second labeling result, the network parameters of the second neural network model are adjusted until the second neural network model converges to obtain the scoring model.
[0061] In some examples, the theoretical node's operational information can be substituted into the scoring formula to obtain the theoretical score. For instance, operational information includes CPU utilization, instruction cache, data cache, and network parameters. When network parameters include speed, bandwidth, throughput, latency, utilization, and packet loss rate, the scoring formula includes: ; .
[0062] Where y represents the theoretical score, a represents the CPU utilization rate, b represents the instruction cache, and c represents the data cache. This represents the score of the network parameters. e represents rate, f represents bandwidth, h represents latency, i represents utilization, and j represents packet loss rate.
[0063] S133. Based on the first score and the second score, determine the target node.
[0064] As described above, the data processing method provided in this disclosure includes: obtaining creation information for creating a container group and running information of each working node; determining the actual graphics processing unit (GPU) memory corresponding to the base model; for working nodes that do not have an actual model identical to the base model, determining one or more first nodes based on the actual GPU memory, dedicated GPU memory, and remaining GPU memory; for working nodes that have an actual model identical to the base model, determining one or more second nodes based on dedicated GPU memory and remaining GPU memory; obtaining a first score for the first node and a second score for the second node; determining a target node based on the first and second scores, and binding the container group to the target node; wherein the target node includes any one of the working nodes; when an actual model identical to the base model exists, obtaining the container identifier of the base model, and the environment variables and mounted volume of the actual model identical to the base model; and creating a container group based on the container identifier, environment variables, mounted volume, creation information, dedicated GPU memory, and the actual model identical to the base model. In this way, when creating a new container group, if the target node has an actual model that is the same as the base model corresponding to the container group, the data processing device can use the actual model, container identifier, environment variables, mounted volume, creation information, and dedicated graphics processor memory that are the same as the base model corresponding to the container group to create the container group. This achieves the reuse of the graphics processor memory of the target node and improves the resource utilization of the graphics processor memory.
[0065] In some feasible examples, combining Figure 3 ,like Figure 4 As shown, the above S130 can be specifically implemented through the following S1300 and S1301.
[0066] S1300: Determine the theoretical graphics processor memory based on actual graphics processor memory and dedicated graphics processor memory; S1301. For working nodes that do not have the same actual model as the base model, if the remaining graphics processor memory is greater than the theoretical graphics processor memory, the working node with the remaining graphics processor memory greater than the theoretical graphics processor memory shall be taken as the first node.
[0067] As described above, the data processing method provided in this embodiment obtains creation information for creating a container group and running information of each working node; determines the actual graphics processing unit (GPU) memory corresponding to the base model; determines the theoretical GPU memory based on the actual GPU memory and the dedicated GPU memory; for working nodes that do not have an actual model identical to the base model, if the remaining GPU memory is greater than the theoretical GPU memory, the working node with the remaining GPU memory greater than the theoretical GPU memory is designated as the first node; for working nodes that have an actual model identical to the base model, one or more second nodes are determined based on the dedicated GPU memory and the remaining GPU memory; obtains a first score for the first node and a second score for the second node; determines a target node based on the first score and the second score, and binds the container group to the target node; wherein, the target node includes any one of the working nodes; when an actual model identical to the base model exists, obtains the container identifier of the base model, as well as the environment variables and mounted volume of the actual model identical to the base model; and creates a container group based on the container identifier, environment variables, mounted volume, creation information, dedicated GPU memory, and the actual model identical to the base model. In this way, when creating a new container group, if the target node has an actual model that is the same as the base model corresponding to the container group, the data processing device can use the actual model, container identifier, environment variables, mounted volume, creation information, and dedicated graphics processor memory that are the same as the base model corresponding to the container group to create the container group. This achieves the reuse of the graphics processor memory of the target node and improves the resource utilization of the graphics processor memory.
[0068] In some feasible examples, combining Figure 3 ,like Figure 5 As shown, the above S131 can be specifically implemented through the following S1310.
[0069] S1310. For a working node that has an actual model that is the same as the base model, if the remaining graphics processor memory is greater than the dedicated graphics processor memory, the working node with the remaining graphics processor memory greater than the dedicated graphics processor memory shall be regarded as the second node.
[0070] As described above, the data processing method provided in this embodiment obtains creation information for creating a container group and running information of each working node; determines the actual graphics processing unit (GPU) memory corresponding to the base model; for working nodes that do not have an actual model identical to the base model, determines one or more first nodes based on the actual GPU memory, dedicated GPU memory, and remaining GPU memory; for working nodes that have an actual model identical to the base model, when the remaining GPU memory is greater than the dedicated GPU memory, the working node with the greater remaining GPU memory is designated as the second node; obtains a first score for the first node and a second score for the second node; determines a target node based on the first and second scores, and binds the container group to the target node; wherein, the target node includes any one of the working nodes; when an actual model identical to the base model exists, obtains the container identifier of the base model, as well as the environment variables and mounted volume of the actual model identical to the base model; and creates a container group based on the container identifier, environment variables, mounted volume, creation information, dedicated GPU memory, and the actual model identical to the base model. In this way, when creating a new container group, if the target node has an actual model that is the same as the base model corresponding to the container group, the data processing device can use the actual model, container identifier, environment variables, mounted volume, creation information, and dedicated graphics processor memory that are the same as the base model corresponding to the container group to create the container group. This achieves the reuse of the graphics processor memory of the target node and improves the resource utilization of the graphics processor memory.
[0071] In some feasible examples, combining Figure 3 ,like Figure 6 As shown, the above S132 can be specifically implemented through the following S1320 and S1321.
[0072] S1320. Obtain the operating information of each theoretical node; wherein, the theoretical node includes either the first node or the second node, and the operating information includes one or more of the following: CPU utilization, instruction cache, data cache, and network parameters; the network parameters include one or more of the following: speed, bandwidth, throughput, latency, utilization, and packet loss rate. S1321. Based on the operational information, determine the theoretical score of each theoretical node; wherein the theoretical score includes either the first score or the second score.
[0073] As described above, the data processing method provided in this embodiment obtains creation information for creating container groups and runtime information of each working node; determines the actual graphics processing unit (GPU) memory corresponding to the base model; for working nodes that do not have an actual model identical to the base model, determines one or more first nodes based on the actual GPU memory, dedicated GPU memory, and remaining GPU memory; for working nodes that have an actual model identical to the base model, determines one or more second nodes based on dedicated GPU memory and remaining GPU memory; and obtains runtime information of each theoretical node; wherein, a theoretical node includes either the first node or the second node, and the runtime information includes CPU utilization, instruction cache, etc. The system uses data caching and one or more network parameters, including one or more of rate, bandwidth, throughput, latency, utilization, and packet loss rate. Based on operational information, it determines the theoretical score of each theoretical node, where the theoretical score includes either a first score or a second score. Based on the first and second scores, it determines the target node and binds the container group to the target node, where the target node includes any of the worker nodes. When an actual model identical to the base model exists, it obtains the container identifier of the base model, as well as the environment variables and mounted volume of the actual model identical to the base model. Based on the container identifier, environment variables, mounted volume, creation information, dedicated graphics processor memory, and the actual model identical to the base model, it creates a container group. Thus, when creating a new container group, if an actual model identical to the base model corresponding to the container group exists in the target node, it can use the actual model, container identifier, environment variables, mounted volume, creation information, and dedicated graphics processor memory corresponding to the base model to create the container group, thereby reusing the graphics processor memory of the target node and improving the resource utilization of the graphics processor memory.
[0074] In some feasible examples, combining Figure 3 ,like Figure 7 As shown, the above S133 can be specifically implemented through the following S1330.
[0075] S1330. Based on the first score and the second score, the working node corresponding to the largest theoretical score is taken as the target node.
[0076] In some examples, when multiple worker nodes have the highest theoretical score, a worker node can be randomly selected from the worker nodes corresponding to the highest scores as the target node, or the total number of actual models run by each worker node corresponding to the highest scores can be obtained, and the worker node corresponding to the smallest total number can be selected as the target node.
[0077] As described above, the data processing method provided in this disclosure includes: obtaining creation information for creating a container group and running information of each working node; determining the actual graphics processing unit (GPU) memory corresponding to the base model; for working nodes that do not have an actual model identical to the base model, determining one or more first nodes based on the actual GPU memory, dedicated GPU memory, and remaining GPU memory; for working nodes that have an actual model identical to the base model, determining one or more second nodes based on dedicated GPU memory and remaining GPU memory; obtaining a first score for the first node and a second score for the second node; based on the first and second scores, selecting the working node corresponding to the highest theoretical score as the target node and binding the container group to the target node; wherein, the target node includes any one of the working nodes; when an actual model identical to the base model exists, obtaining the container identifier of the base model, and the environment variables and mounted volume of the actual model identical to the base model; and creating a container group based on the container identifier, environment variables, mounted volume, creation information, dedicated GPU memory, and the actual model identical to the base model. In this way, when creating a new container group, if the target node has an actual model that is the same as the base model corresponding to the container group, the data processing device can use the actual model, container identifier, environment variables, mounted volume, creation information, and dedicated graphics processor memory that are the same as the base model corresponding to the container group to create the container group. This achieves the reuse of the graphics processor memory of the target node and improves the resource utilization of the graphics processor memory.
[0078] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0079] This application embodiment can divide the data processing device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0080] like Figure 8 As shown in the diagram, an embodiment of this application provides a schematic diagram of a data processing device. It includes a communicator 101 and a controller 102.
[0081] The communicator 101 is configured to: acquire creation information for creating a container group and running information for each worker node; wherein, the creation information includes: the basic model corresponding to the container group and the dedicated graphics processor memory of the container group, and the running information includes the remaining graphics processor memory and one or more currently running actual models. Controller 102 is configured as follows: Determine the actual graphics processor memory corresponding to the base model; Based on the actual graphics processor memory, basic model, dedicated graphics processor memory, and runtime information, the target node is determined, and the container group is bound to the target node; wherein, the target node includes any one of the worker nodes; If an actual model with the same characteristics as the base model exists, obtain the container identifier of the base model, as well as the environment variables and mounted volumes of the actual model with the same characteristics as the base model. Create a container group based on container identifiers, environment variables, mounted volumes, creation information, dedicated graphics processor memory, and an actual model identical to the base model.
[0082] In some implementable examples, when the controller 102 determines the target node based on the actual graphics processor memory, the base model, the dedicated graphics processor memory, and runtime information, it is further configured to: For working nodes that do not have the same actual model as the base model, one or more first nodes are determined based on the actual graphics processor memory, dedicated graphics processor memory, and remaining graphics processor memory. For working nodes that have the same actual model as the base model, one or more second nodes are determined based on the dedicated graphics processor memory and the remaining graphics processor memory; Get the first score of the first node and the second score of the second node; The target node is determined based on the first and second scores.
[0083] In some implementable examples, when the controller 102 determines one or more first nodes based on the actual graphics processor memory, dedicated graphics processor memory, and remaining graphics processor memory for a working node that does not have an actual model identical to the base model, it is further configured to: The theoretical graphics processor memory is determined based on actual graphics processor memory and dedicated graphics processor memory. For working nodes that do not have the same actual model as the base model, if the remaining GPU memory is greater than the theoretical GPU memory, the working node with the remaining GPU memory greater than the theoretical GPU memory is taken as the first node.
[0084] In some implementable examples, when the controller 102 determines one or more second nodes based on dedicated graphics processor memory and remaining graphics processor memory for a working node that has an actual model identical to the base model, it is further configured to: For a working node that has the same actual model as the base model, if the remaining GPU memory is greater than the dedicated GPU memory, the working node with the greater remaining GPU memory is designated as the second node.
[0085] In some implementable examples, controller 102 is further configured to, when performing the acquisition of the first score of the first node and the second score of the second node: Obtain the operational information of each theoretical node; wherein, the theoretical node includes either the first node or the second node, and the operational information includes one or more of the following: CPU utilization, instruction cache, data cache, and network parameters; the network parameters include one or more of the following: speed, bandwidth, throughput, latency, utilization, and packet loss rate. Based on the operational information, the theoretical score for each theoretical node is determined; the theoretical score includes either the first score or the second score.
[0086] In some implementable examples, controller 102 is further configured to determine the target node based on a first score and a second score as follows: Based on the first and second scores, the working node corresponding to the highest theoretical score is taken as the target node.
[0087] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and their functions will not be repeated here.
[0088] Of course, the data processing device provided in this application embodiment includes, but is not limited to, the modules described above. For example, the data processing device may also include a memory 103. The memory 103 may be used to store the program code of the data processing device, and may also be used to store data generated by the data processing device during operation, such as data in write requests.
[0089] like Figure 9As shown, this application embodiment also provides a chip system that can be applied to the data processing device in the foregoing embodiments. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 may be the processor in the aforementioned data processing device. The processor 1501 and the interface circuit 1502 are interconnected via a line. The processor 1501 can receive and execute computer instructions from the memory of the aforementioned data processing device through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the data processing device can perform the various steps executed by the data processing device in the foregoing embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.
[0090] This application also provides a computer-readable storage medium for storing computer instructions for operating the aforementioned data processing device.
[0091] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing device, characterized in that, include: The communicator is configured to: acquire creation information for creating a container group and runtime information for each worker node; wherein the creation information includes: the base model corresponding to the container group and the dedicated graphics processor memory of the container group, and the runtime information includes the remaining graphics processor memory and one or more currently running actual models; The controller is configured as follows: Determine the actual graphics processor memory corresponding to the basic model; Based on the actual graphics processor memory, the base model, the dedicated graphics processor memory, and the runtime information, a target node is determined, and the container group is bound to the target node; wherein, the target node includes any one of the working nodes; If an actual model identical to the base model exists, obtain the container identifier of the base model, as well as the environment variables and mounted volumes of the actual model identical to the base model. The container group is created based on the container identifier, the environment variables, the mounted volume, the creation information, the dedicated graphics processor memory, and the actual model that is the same as the base model.
2. The data processing device according to claim 1, characterized in that, When the controller determines the target node based on the actual graphics processor memory, the base model, the dedicated graphics processor memory, and the runtime information, it is further configured to: For working nodes that do not have an actual model identical to the base model, one or more first nodes are determined based on the actual graphics processor memory, the dedicated graphics processor memory, and the remaining graphics processor memory. For a working node that has an actual model identical to the base model, one or more second nodes are determined based on the dedicated graphics processor memory and the remaining graphics processor memory. Obtain the first score of the first node and the second score of the second node; The target node is determined based on the first score and the second score.
3. The data processing device according to claim 2, characterized in that, When the controller executes a work node for which there is no actual model identical to the base model, and determines one or more first nodes based on the actual graphics processor memory, the dedicated graphics processor memory, and the remaining graphics processor memory, it is further configured to: Based on the actual graphics processor memory and the dedicated graphics processor memory, the theoretical graphics processor memory is determined. For a working node that does not have an actual model identical to the base model, when the remaining graphics processor memory is greater than the theoretical graphics processor memory, the working node with the remaining graphics processor memory greater than the theoretical graphics processor memory is designated as the first node.
4. The data processing device according to claim 2, characterized in that, When the controller, in executing a work node that has an actual model identical to the base model, determines one or more second nodes based on the dedicated graphics processor memory and the remaining graphics processor memory, it is further configured to: For a working node that has an actual model identical to the base model, if the remaining GPU memory is greater than the dedicated GPU memory, the working node with the remaining GPU memory greater than the dedicated GPU memory is designated as the second node.
5. The data processing device according to claim 2, characterized in that, When the controller executes the process of obtaining the first score of the first node and the second score of the second node, it is further configured to: Obtain the operational information of each theoretical node; wherein, the theoretical node includes either the first node or the second node, and the operational information includes one or more of the following: CPU utilization, instruction cache, data cache, and network parameters, and the network parameters include one or more of the following: speed, bandwidth, throughput, latency, utilization, and packet loss rate; Based on the operational information, the theoretical score of each theoretical node is determined; wherein the theoretical score includes either a first score or a second score.
6. The data processing device according to claim 2, characterized in that, When the controller determines the target node based on the first score and the second score, it is further allocated as follows: Based on the first score and the second score, the working node corresponding to the highest theoretical score is taken as the target node.
7. A data processing method, characterized in that, include: Obtain creation information for creating container groups and runtime information for each worker node; wherein, the creation information includes: the base model corresponding to the container group and the dedicated graphics processor memory of the container group, and the runtime information includes the remaining graphics processor memory and one or more currently running actual models; Determine the actual graphics processor memory corresponding to the basic model; Based on the actual graphics processor memory, the base model, the dedicated graphics processor memory, and the runtime information, a target node is determined, and the container group is bound to the target node; wherein, the target node includes any one of the working nodes; If an actual model identical to the base model exists, obtain the container identifier of the base model, as well as the environment variables and mounted volumes of the actual model identical to the base model. The container group is created based on the container identifier, the environment variables, the mounted volume, the creation information, the dedicated graphics processor memory, and the actual model that is the same as the base model.
8. The data processing method according to claim 7, characterized in that, The process of determining the target node based on the actual graphics processor memory, the basic model, the dedicated graphics processor memory, and the runtime information includes: For working nodes that do not have an actual model identical to the base model, one or more first nodes are determined based on the actual graphics processor memory, the dedicated graphics processor memory, and the remaining graphics processor memory. For a working node that has an actual model identical to the base model, one or more second nodes are determined based on the dedicated graphics processor memory and the remaining graphics processor memory. Obtain the first score of the first node and the second score of the second node; The target node is determined based on the first score and the second score.
9. The data processing method according to claim 8, characterized in that, For working nodes that do not have an actual model identical to the base model, one or more first nodes are determined based on the actual graphics processor memory, the dedicated graphics processor memory, and the remaining graphics processor memory, including: Based on the actual graphics processor memory and the dedicated graphics processor memory, the theoretical graphics processor memory is determined. For a working node that does not have an actual model identical to the base model, when the remaining graphics processor memory is greater than the theoretical graphics processor memory, the working node with the remaining graphics processor memory greater than the theoretical graphics processor memory is designated as the first node.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method as described in claim 7 or 8.