Generation method for host pcie topological structure and container resource allocation method
By constructing the PCIe topology of the host, the nearest matching of container resources is achieved, and the affinity problem of resource allocation in the container orchestration system is solved, and computing performance and efficiency are improved.
Patent Information
- Application Number
- PCT/CN2024/088378
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-04-17
- Publication Date
- 2024-11-14
AI Technical Summary
Container orchestration systems such as K8s are difficult to obtain the topological relationships of host computing, network and other components, resulting in affinity problems in the resources allocated by the container, increasing latency and resource waste.
By obtaining the host's PCIe topology, including the association relationship between network cards, GPUs and NUMA nodes, a multi-layer topology structure of host resources is constructed, and the layer-by-layer near-matching of CPUs, memory, network cards, GPUs and their virtualization instances are achieved, and container resource allocation is performed. .
It effectively solves the unnecessary data transmission and resource waste problems of containers when performing computing tasks, and improves the computing performance and resource utilization efficiency of containers.
Smart Images

Figure CN2024088378_14112024_PF_FP_ABST
Abstract
Description
Host PCIe topology generation method and container resource allocation method
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 11, 2023, with application number 2023105313887 and invention name “Method for generating host PCIe topology structure and container resource allocation method”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of computer technology, and in particular to a method for generating a host PCIe topology structure and a method for allocating container resources. Background Art
[0003] In recent years, the theory and technology of artificial intelligence have matured, and its application areas have continued to expand. This trend has gradually shifted from one-way product supply to deep, two-way collaboration across industries. Cloud service platforms have become a crucial channel for enterprises to acquire and apply AI capabilities. Cloud-native architectures, with their advantages of elastic scalability, agile distribution, high efficiency, ease of use, and compatibility, have gained widespread industry recognition, helping users reduce cloud costs and improve the availability and quality of cloud services. Container orchestration technologies, such as Kubernetes, create containers like Docker that are more lightweight, agile, and efficient than virtual machines, and have become a key foundational technology for cloud-native computing.
[0004] Virtualization, as a new technology, can map physical network cards, graphics processing units (GPUs), etc. to one or more virtual functions, and support finer-grained allocation of them in containers through virtualized instances, thereby improving resource utilization efficiency. However, in current resource pools, since container orchestration systems such as K8s find it difficult to obtain the topological relationship between the host's processor (Central Processing Unit, CPU), memory, network cards, GPUs and other computing, network and storage components, the resources allocated to containers may have affinity issues, resulting in unnecessary data transmission when performing computing tasks, increased latency and resource waste. And because virtualization generates finer-grained resources, this problem becomes more prominent compared to the allocation of traditional physical resources.
[0005] For the above problems, no effective solutions have been proposed yet.
[0006] Summary of the Invention
[0007] The embodiments of the present application provide a method for generating a host PCIe topology structure and a method for allocating container resources, which at least solve the technical problem in the related art that, since container orchestration systems such as K8s have difficulty in obtaining the topological relationships of host computing, network and other components, their allocated resources may have affinity allocation problems, resulting in unnecessary data transmission when the container performs computing tasks, causing increased latency and waste of resources.
[0008] According to one aspect of an embodiment of the present application, a method for generating a host PCIe topology structure is provided, comprising: obtaining a first topology structure and a second topology structure respectively; wherein the first topology structure is used to represent a topology structure formed by a network card, a network card virtual function, a graphics processor, and a virtual graphics processor associated with a host under a PCIe adapter unit; the second topology structure is used to represent a topology structure formed by a CPU interface unit, a CPU core, and a memory associated with a non-uniform memory access (NUMA) node based on the host; obtaining an association relationship between the PCIe adapter unit and the NUMA node; and associating the first topology structure and the second topology structure based on the association relationship to obtain the PCIe topology structure of the host.
[0009] According to another aspect of an embodiment of the present application, a container resource allocation method is provided, comprising: obtaining proposed allocated resources for a container; and when allocable resources of a host satisfy the proposed allocated resources, allocating resources to the container based on a PCIe topology of the host, wherein the PCIe topology of the host is obtained based on any one of the methods described above.
[0010] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes any one of the methods described above.
[0011] According to another aspect of an embodiment of the present application, a non-transitory machine-readable medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are only some embodiments of this application. For those of ordinary skill in the art, other embodiments can be derived from these drawings without inventive effort.
[0013] FIG1 is a schematic diagram of a state of randomly allocating container resources within a host according to an embodiment of the present application;
[0014] FIG2 is a flow chart of a method for generating a host PCIe topology structure according to an embodiment of the present application;
[0015] FIG3 is a schematic diagram of a host PCIe topology and resource proximity allocation provided by an embodiment of the present application;
[0016] FIG4 is a schematic diagram of a host PCIe topology generation process according to an embodiment of the present application;
[0017] FIG5 is a flowchart of a container resource allocation method provided in an embodiment of the present application;
[0018] FIG6 is a schematic diagram of a container resource allocation process provided by an embodiment of the present application;
[0019] FIG7 is a schematic diagram of a host PCIe topology structure and resource proximity allocation under a single NUMA node provided by an embodiment of the present application;
[0020] FIG8 shows a hardware structure block diagram of an electronic device. DETAILED DESCRIPTION
[0021] The following describes embodiments of the present invention in more detail with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.
[0022] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:
[0023] K8s: Kubernetes (also known as k8s or "kube") is an open source container orchestration platform that automates many of the manual steps involved in deploying, managing, and scaling containerized applications.
[0024] Kubelet: A service running on a K8s node that reads the container manifest and ensures that the specified container is up and running.
[0025] Containers: Containers are a form of operating system virtualization. You can use a single container to run everything from small microservices or software processes to large applications. Unlike server or computer virtualization methods, containers do not contain an operating system image, making them more lightweight and portable. In large application deployments, multiple containers can be deployed as one or more container clusters.
[0026] Virtualization: Computer virtualization technology is a resource management technology that abstracts and converts various physical resources of a computer (CPU, GPU, memory, network card, etc.) into resource instances that can be divided and recombined, thereby breaking down physical structural barriers and allowing users to use computer resources in a better way than the original configuration.
[0027] SR-IOV: Single Root I / O Virtualization is an extension to the PCIe specification that allows devices (such as network adapters) to separate access to their resources between various PCIe hardware functions.
[0028] VF: Virtual Functions, a lightweight PCIe function on a network adapter that supports Single Root I / O Virtualization (SR-IOV). A VF is associated with a PCIe physical function (PF) on the network adapter and represents a virtualized instance of the network adapter.
[0029] PCIe: PCI-Express (Peripheral Component Interconnect Express) is a high-speed serial computer expansion bus standard, which belongs to high-speed serial point-to-point dual-channel high-bandwidth transmission.
[0030] Topology: Topology is a discipline that studies the properties of geometric figures or spaces that remain unchanged after continuous changes in shape. It only considers the positional relationship between objects without considering their shape and size. Computer topology refers to the description of various physical layout features by borrowing the two most basic graphic elements in geometry, points and lines, to abstractly discuss the methods, forms and geometric shapes of the interconnection of various components in the computer system.
[0031] PCIe Switch: A PCIe to PCIe bridge that provides expansion or aggregation capabilities and allows more devices to be connected to a PCIe port.
[0032] NUMA / Numa Node: Non-Uniform Memory Access (NUM) is a computer memory design for multi-processors. In a NUMA node, a processor can access its local memory faster than non-local memory.
[0033] CPU Socket: The CPU socket is an interface for installing the CPU, usually consisting of some metal pins and slots, used to connect the CPU and the motherboard.
[0034] RDMA: Remote Direct Memory Access is a technology that implements zero-copy and kernel bypass technology through hardware to provide high-performance network data access.
[0035] In one implementation, in the default Kubernetes resource allocation scheme, the Device Plugin first registers and reports the node's resources to the kubelet at startup. The kubelet then reports the node's resource quantities to the Kubernetes scheduler. When creating a container, Kubernetes selects a node that meets the resource requirements and sends the required resource quantities to the corresponding kubelet. The kubelet then allocates the corresponding resources directly from the recorded device list in order. Because the order of resource reporting and container creation is random, the default allocation scheme effectively results in a random allocation of resources.
[0036] Under the random allocation strategy, the affinity of resources such as CPU, memory, network interface cards (NIC, VF), and graphics cards (GPU, vGPU) within the container cannot be determined, and there are many possibilities. For example, some possible allocation situations in a virtualized environment are shown in Figure 1:
[0037] (1) The container's VF and vGPU belong to the same PCIe switch, and the CPU and memory belong to the same NUMA node, as shown in the gray block in Figure 1. This is consistent with the affinity feature, which can minimize unnecessary data transmission, improve the computing efficiency of tasks within the container, and reduce costs.
[0038] (2) The container's VF and vGPU belong to different PCIe switches under the same NUMA node, as shown in the left-slanted color block in Figure 1. In this case, unnecessary cross-PCIe switch data transmission occurs between the VF and vGPU, resulting in performance degradation.
[0039] (3) The container's VF and vGPU belong to different NUMA nodes under the same CPU socket, as shown in the right-slashed color block in Figure 1. In this case, unnecessary cross-NUMA node data transmission occurs between the VF and vGPU, further degrading performance.
[0040] (4) The container's VF and vGPU belong to different CPU sockets, as shown by the cross-hatched blocks in Figure 1, or the container's CPU, memory, VF, and vGPU belong to different CPU sockets, as shown by the dotted blocks in Figure 1. In this case, unnecessary cross-CPU socket data transmission occurs between the VF and vGPU, or between the VF, vGPU, CPU, and memory, further impacting performance.
[0041] In virtualized scenarios, due to the finer granularity of resources, affinity issues are more prominent during random allocation than in traditional physical resource scenarios. On the one hand, the increase in resource instances reduces the probability that allocation results under random policies will meet affinity characteristics. On the other hand, the relatively small number of resources in a single virtualized instance exacerbates bandwidth contention and performance degradation caused by additional transmission.
[0042] The current Kubernetes implementation provides a mechanism for resource affinity allocation based on NUMA nodes. However, since affinity support is limited to the NUMA node layer, this policy is completely ineffective for single-NUMA node machines. Furthermore, this policy cannot achieve affinity at the PCIe switch layer, nor can it achieve affinity at the CPU socket layer when the same-NUMA node condition is not met. Furthermore, in virtualized scenarios, with a large number of VFs and vGPUs and a small number of individual resources, more fine-grained proximity allocation is urgently needed, resulting in significant limitations for this policy.
[0043] To overcome the above shortcomings, an embodiment of the present application provides a method for generating a host PCIe topology structure. FIG2 is a flow chart of the method for generating a host PCIe topology structure provided by an embodiment of the present application. As shown in FIG2 , the method includes the following steps:
[0044] Step S202: Obtain a first topology and a second topology, respectively; wherein the first topology is used to represent a topology formed by a network interface card (NIC), a network interface card virtual function (NIC virtual function), a graphics processor (GPU), and a virtual graphics processor (VGPU) associated with a host under a PCIe adapter unit; and the second topology is used to represent a topology formed by a CPU interface unit (CPU), a CPU core, and a memory associated with a NUMA node.
[0045] The CPU core is also called a CPU Core; a VF is a virtualized instance of a network card; and a vGPU is a virtualized instance of a graphics processing unit. Furthermore, in one implementation, a PCIe adapter unit is used to connect a PCIe bus to multiple PCIe devices, enabling data exchange and communication between multiple devices. PCIe adapter units include, but are not limited to, PCIe switches. A CPU interface unit is used to connect the CPU and motherboard. CPU interface units include, but are not limited to, CPU sockets.
[0046] Step S204, obtaining the association relationship between the PCIe adapter unit and the NUMA node; Step S206, associating the first topology structure and the second topology structure based on the association relationship to obtain the PCIe topology structure of the host.
[0047] The PCIe topology of the host is also called the multi-layer topology of host resources, which is used to describe the topological relationship between host computing, network, and storage resources.
[0048] In an embodiment of the present application, by constructing a multi-layer topology structure of host resources from the CPU interface unit, NUMA node to the PCIe adapter unit, the layer-by-layer proximity matching of the CPU, memory, network card, GPU and their virtualized instances is achieved, thereby solving the technical problem in the related technology that the container orchestration system such as K8s is difficult to obtain the topological relationship of the host computing, network and other components, and the allocated resources may have affinity allocation problems, resulting in unnecessary data transmission when the container performs computing tasks, resulting in increased latency and waste of resources. The technical effect is achieved that can largely solve the affinity allocation problem of resources in the container, avoid unnecessary data transmission, and improve the computing performance of the container.
[0049] As an embodiment, in a specific implementation process, obtaining the first topology structure includes the following steps: Step 21, obtaining the PCIe path of the network card, the NUMA node to which the network card belongs, the network card virtual function, and the mapping relationship between the network card and the network card virtual function; Step 22, obtaining the PCIe path of the graphics processor, the NUMA node to which the graphics processor belongs, the virtual graphics processor, and the mapping relationship between the graphics processor and the virtual graphics processor; Step 23, based on the PCIe path of the network card and the PCIe path of the graphics processor, identifying the network card and the graphics processor under the same PCIe adapter unit;
[0050] Step 24: Acquire a first topology based on the NUMA node to which the network card belongs, the network card virtual function, the mapping relationship between the network card and the network card virtual function, the NUMA node to which the graphics processor belongs, the virtual graphics processor, the mapping relationship between the graphics processor and the virtual graphics processor, and the association relationship between the network card and the graphics processor under the same PCIe adapter unit.
[0051] A PCIe lane refers to the data transmission path on the PCI Express bus. The PCIe bus is a high-speed serial bus that transmits data through multiple channels (called "lanes"). Each lane has a pair of differential signal lines for transmitting data and clock signals. A PCIe path typically consists of one or more transmission paths, each with a separate data transmission path. Data can be interleaved between lanes to increase bus width. A PCIe path also includes devices and interfaces connected to the bus that can transfer data from one lane to another. For example, the PCIe path can be "pci0000:17 / 0000:17:00.0 / 0000:18:00.0 / 0000:19:04.0", where "pci0000:17" indicates that the device is connected to the bus with PCI bus number 17; "0000:17:00.0" indicates that the device is the first device on PCI bus 17 and has a function number of 0; "0000:18:00.0" indicates that the PCI bus number connected to the upper-level device is 18, the device number is 0, and the function number is 0; and "0000:19:04.0" indicates that the PCI bus number connected to the upper-level device is 19, the device number is 4, and the function number is 0.
[0052] The above PCIe paths use Bus / Device / Function (BDF) numbers to describe PCI and PCIe devices, including hexadecimal bus numbers, device numbers, and function numbers.
[0053] In one embodiment, the above-mentioned step 24 includes the following specific implementation process: step 2401, based on the association relationship between the network card and the graphics processor under the same PCIe adapter unit, obtaining a first sub-topology structure; step 2402, based on the mapping relationship between the network card and the network card virtual function, associating the network card virtual function with the first sub-topology structure, and associating the NUMA node to which the network card belongs with the first sub-topology structure, to obtain a first target topology structure; step 2403, based on the mapping relationship between the graphics processor and the virtual graphics processor, associating the virtual graphics processor with the first sub-topology structure, and associating the NUMA node to which the graphics processor belongs with the first sub-topology structure, to obtain a second target topology structure; step 2404, associating the first target topology structure and the second target topology structure to obtain a first topology structure.
[0054] In an embodiment of the present application, by accurately obtaining the connection mode and topology of computer hardware (the topology formed by the network card, network card virtual function, graphics processor, and virtual graphics processor associated under the PCIe adapter unit), the performance optimization and resource management of the container can be better performed.
[0055] As an embodiment, in the specific implementation process, obtaining the second topology structure includes the following steps: Step 31, obtaining the CPU interface unit, the NUMA node contained in the CPU interface unit, the CPU core, the NUMA node to which the CPU core belongs, the memory, and the NUMA node to which the memory belongs; Step 32, obtaining the second topology structure based on the association relationship between the CPU interface unit, the NUMA node contained in the CPU interface unit, the CPU core, the NUMA node to which the CPU core belongs, the memory, and the NUMA node to which the memory belongs.
[0056] In one embodiment, the above-mentioned step 32 includes the following specific implementation process: step 3201, based on the association relationship between the CPU interface unit and the NUMA node contained in the CPU interface unit, obtain the second sub-topology structure; step 3202, based on the NUMA node to which the CPU core belongs and the NUMA node to which the memory belongs, respectively associate the CPU core and the memory to the second sub-topology structure to obtain the second topology structure.
[0057] In an embodiment of the present application, by accurately obtaining the connection mode and topology of computer hardware (the topology formed by the CPU interface unit, CPU core and memory associated under the NUMA node), the performance optimization and resource management of the container can be better performed.
[0058] In one embodiment of the present application, the process of first generating the PCIe topology of the resources in the host is shown in Figure 3, and the main steps include: Step 301: read the network card information through the host PCIe structure, or read its PCIe path through the network card file, and obtain the information of the NUMA node to which it belongs. On this basis, read the mapping relationship between the network card and the VF, and establish the association between the network card and the VF; Step 302: read the GPU information through the host PCIe structure, or read its PCIe path through the GPU file, and obtain the information of the NUMA node to which it belongs. On this basis, read the mapping relationship between the GPU and the vGPU, and establish the association between the GPU and the vGPU; Step 303: based on the above steps, identify the network card and GPU under the same PCIe switch through the PCIe path, and construct a sub-topology structure of each PCIe switch and the associated network card, GPU and its virtualized instance; Step 304: read the host CPU and memory information, including CPU Core, CPU Socket, memory and NUMA node data, construct the topology structure of the CPU Socket and NUMA node in the host as the top part of the host PCIe topology, and associate the CPU Core and memory; Step 305: By matching NUMA node information, the PCIe Switch sub-topology structure generated in Step 3 is associated with the CPU Socket and NUMA node top-level topology structure generated in Step 4 to generate a complete topology structure.
[0059] It should be noted that the above-mentioned host PCIe structure includes but is not limited to the CPU Socket layer, the NUMA node layer and the PCIe Switch layer, wherein each layer includes one or more nodes, each node represents a component of the layer, and the connection between the nodes represents the association relationship between the corresponding components of each node.
[0060] The host PCIe topology structure obtained by this solution is shown in Figure 4. In actual use, it can also be represented in a structured manner such as JSON.
[0061] In response to the lack of reference information for K8s container resource allocation, this solution first generates a multi-layer PCIe topology for the host, solving the problem of unclear basis for allocation optimization. On this basis, in order to address the affinity issues that may be caused by the default random scheduling strategy of K8s, this solution, based on the generated PCIe topology, can achieve layer-by-layer proximity matching of network, computing and other resources, thereby avoiding unnecessary transmission caused by container resource affinity issues to the greatest extent, thereby improving efficiency and reducing communication costs. Compared with the new version of K8s's NUMA node-based scheduling strategy, this solution can not only adapt to hosts with a single NUMA node structure, but also achieve more fine-grained proximity allocation at the PCIe Switch and CPU Socket layers in a multi-NUMA node environment, achieving better affinity effects.
[0062] In virtualization scenarios with more resource instances and finer granularity, this solution can significantly increase the probability of affinity allocation compared to the default K8s strategy. By avoiding additional transmission, it can further focus limited resources on actual tasks, thereby significantly improving container performance.
[0063] According to another aspect of an embodiment of the present application, a container resource allocation method is provided. Figure 5 is a flowchart of the container resource allocation method provided in an embodiment of the present application. As shown in Figure 5, the method includes the following steps: Step S502, obtaining the container's proposed allocated resources; the above-mentioned proposed allocated resources include but are not limited to the CPU, memory, network card virtual function and virtual graphics processor required by the container.
[0064] Step S504 : when the allocatable resources of the host satisfy the intended allocatable resources, allocate resources to the container based on the PCIe topology of the host, wherein the PCIe topology of the host is obtained based on any of the above methods.
[0065] The aforementioned allocatable resources include, but are not limited to, the remaining CPUs, memories, network card virtual functions, and virtual graphics processors of each NUMA node.
[0066] In an embodiment of the present application, by constructing a multi-layer topology structure of host resources from the CPU interface unit, NUMA node to the PCIe adapter unit, the layer-by-layer proximity matching of the CPU, memory, network card, GPU and their virtualized instances is achieved, thereby solving the technical problem in the related technology that the container orchestration system such as K8s is difficult to obtain the topological relationship of the host computing, network and other components, and the allocated resources may have affinity allocation problems, resulting in unnecessary data transmission when the container performs computing tasks, resulting in increased latency and waste of resources. The technical effect is achieved that can largely solve the affinity allocation problem of resources in the container, avoid unnecessary data transmission, and improve the computing performance of the container.
[0067] As an embodiment, in a specific implementation process, resource allocation for a container based on the PCIe topology of the host includes the following steps: Step 41, determining a NUMA node whose remaining resources are greater than preset remaining resources as a target NUMA node; in one embodiment, the PCIe topology of the host includes multiple NUMA nodes, and the NUMA node with the most remaining resources can be screened out from the multiple NUMA nodes, and the NUMA node with the most remaining resources can be used as the target NUMA node.
[0068] Step 42: Allocate allocatable resources to the container based on the target NUMA node.
[0069] In an embodiment of the present application, resources are allocated to containers by utilizing a NUMA node whose remaining resources are greater than preset remaining resources.
[0070] As an embodiment, when the allocatable resource is a CPU, the above step 42 includes the following specific implementation process: step 4201, judging whether there are idle CPU cores in the target NUMA node and whether the number of idle CPU cores meets the number of CPU cores required by the container; step 4202, if there are idle CPU cores in the target NUMA node and the number of idle CPU cores meets the number of CPU cores required by the container, then allocating the idle CPU cores to the container; step 4203, if there are no idle CPU cores in the target NUMA node or the number of idle CPU cores does not meet the number of CPU cores required by the container, then allocating the container to a CPU interface unit that is in the same CPU interface unit as the target NUMA node. CPU core; In one embodiment, allocating CPU cores under the same CPU interface unit as the target NUMA node to the container includes the following specific implementation steps: Step 4204, determining whether the number of CPU cores under the same CPU interface unit meets the number of CPU cores required by the container; Step 4205, if the number of CPU cores under the same CPU interface unit meets the number of CPU cores required by the container, allocating the CPU cores under the same CPU interface unit to the container; Step 4206, if the number of CPU cores under the same CPU interface unit does not meet the number of CPU cores required by the container, allocating the CPU cores under a CPU interface unit different from the target NUMA node to the container.
[0071] In the embodiments of the present application, reasonable and effective allocation of CPU resources can be achieved.
[0072] As an embodiment, when the allocatable resource is memory, the above-mentioned step 42 also includes the following specific implementation process: step 4207, judging whether the remaining memory of the target NUMA node satisfies the memory required by the container; step 4208, if the remaining memory of the target NUMA node satisfies the memory required by the container, the remaining memory of the target NUMA node is allocated to the container; step 4209, if the remaining memory of the target NUMA node does not meet the memory required by the container, the remaining memory of other NUMA nodes under the same CPU interface unit is allocated to the container.
[0073] In one embodiment, allocating the remaining memory of other NUMA nodes under the same CPU interface unit to the container includes the following specific implementation steps: step 4210, determining whether the remaining memory of other NUMA nodes under the same CPU interface unit meets the memory required by the container; step 4211, if the remaining memory of other NUMA nodes meets the memory required by the container, allocating the remaining memory of other NUMA nodes to the container; step 4212, if the remaining memory of other NUMA nodes does not meet the memory required by the container, allocating the memory under a CPU interface unit different from the target NUMA node to the container.
[0074] In the embodiments of the present application, reasonable and effective allocation of memory resources can be achieved.
[0075] As an embodiment, when the allocatable resources are network card virtual functions and virtual graphics processors, before allocatable resources are allocated to containers based on the target NUMA node, the above method further includes the following specific implementation process: Step 52, determining whether the network card virtual functions and virtual graphics processors of the same PCIe adapter unit meet the network card virtual functions and virtual graphics processors required by the container; Step 54, if the network card virtual functions and virtual graphics processors of the same PCIe adapter unit meet the network card virtual functions and virtual graphics processors required by the container, then allocating the network card virtual functions and virtual graphics processors of the same PCIe adapter unit to the container.
[0076] It should be noted that if the network card virtual function and virtual graphics processor of the same PCIe adapter unit do not meet the network card virtual function and virtual graphics processor required by the container, the network card virtual function and virtual graphics processor of the target NUMA node are allocated to the container, that is, the content related to step 42 is executed.
[0077] As an embodiment, when the allocatable resources are network card virtual functions and virtual graphics processors, the above-mentioned step 42 also includes the following specific implementation process: step 4213, judging whether the network card virtual functions and virtual graphics processors of the target NUMA node meet the network card virtual functions and virtual graphics processors required by the container; step 4214, if the network card virtual functions and virtual graphics processors of the target NUMA node meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors of the target NUMA node to the container; step 4215, if the network card virtual functions and virtual graphics processors of the target NUMA node do not meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors under the same CPU interface unit as the target NUMA node to the container.
[0078] In one embodiment, allocating network card virtual functions and virtual graphics processors in the same CPU interface unit as the target NUMA node to the container includes the following specific implementation process: Step 4216, determining whether the network card virtual functions and virtual graphics processors in the same CPU interface unit as the target NUMA node meet the network card virtual functions and virtual graphics processors required by the container; Step 4217, if the network card virtual functions and virtual graphics processors in the same CPU interface unit as the target NUMA node meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors in the same CPU interface unit to the container; Step 4218, if the network card virtual functions and virtual graphics processors in the same CPU interface unit as the target NUMA node do not meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors in a different CPU interface unit to the container.
[0079] The network card virtual functions and virtual graphics processors required by the above containers can be reflected in multiple aspects such as quantity, size, and type.
[0080] In the embodiments of the present application, reasonable and effective allocation of VF and vGPU resources can be achieved.
[0081] As an embodiment, the above-mentioned step 41 also includes the following specific implementation process: step 4101, obtaining the remaining resources of the NUMA node and the weight coefficient corresponding to the remaining resources; step 4102, calculating the remaining resource evaluation value of the NUMA node based on the remaining resources and the weight coefficient corresponding to the remaining resources; step 4103, obtaining the preset remaining resource evaluation value corresponding to the preset remaining resources; step 4104, when the remaining resource evaluation value is greater than the preset remaining resource evaluation value, determining that the remaining resources of the NUMA node are greater than the preset remaining resources, and taking the NUMA node as the target NUMA node.
[0082] The above-mentioned remaining resources and preset remaining resources can both be evaluated using the remaining resource assessment value.
[0083] The aforementioned remaining resources are the allocatable resources corresponding to the NUMA node, including but not limited to the remaining CPUs, memory, network card virtual functions, and virtual graphics processors of each NUMA node. During implementation, corresponding weight coefficients can be set for the remaining resources. For example, if the weight coefficient for the CPU is 0.3, the weight coefficient for the memory is 0.3, the weight coefficient for the network card virtual function is 0.2, and the weight coefficient for the virtual graphics processor is 0.2, then the remaining resource assessment value of the NUMA node is 0.3*number of CPUs + 0.3*memory size + 0.2*number of network card virtual functions + 0.2*number of virtual graphics processors. It should be noted that the weight coefficients can be flexibly set according to the needs of the application scenario.
[0084] In an embodiment of the present application, the target NUMA node is obtained more accurately by calculating and comparing the remaining resource evaluation values.
[0085] As an embodiment, if there are multiple NUMA nodes whose remaining resources are greater than the preset remaining resources, the NUMA node with the largest remaining resource evaluation value is used as the target NUMA node.
[0086] It's important to note that the remaining resource evaluation value is a comprehensive calculation based on the quantity and quality of the NUMA node's remaining CPU, memory, storage, and other resources. A higher evaluation value indicates more abundant remaining resources on the node, better able to meet task requirements. Therefore, selecting the NUMA node with the highest evaluation value as the target node maximizes system resource utilization and improves task execution efficiency and performance.
[0087] In one embodiment of the present application, based on the generated host PCIe topology, container resources can be allocated locally. The process is shown in Figure 6, and the main steps include: Step 601: After kubelet receives the number of resources to be allocated sent by K8s, it determines whether the existing allocatable resources in the host meet the demand. If any resource is insufficient, it directly reports an error and returns; it should be noted that based on the K8s node scheduling mechanism, the kubelet component will periodically report the node's resource status to the application programming interface server (API Server). When the node receives the demand for resources to be allocated, K8s has already pre-judged that the total amount of node resources can meet the demand.
[0088] Step 602: Select the NUMA node with the most remaining resources for pre-allocation; Step 603: Allocate CPU: First select the free CPU Core in the same NUMA node. If the number is insufficient, then select the CPU Core in the same CPU Socket and across CPU Sockets in turn; Step 604: Allocate memory: If the NUMA node has enough memory, select it; otherwise, try to select memory in the same CPU Socket and across CPU Sockets in turn; Step 605: Allocate VF and vGPU: Traverse each PCIe Switch in the NUMA node. If there are enough VFs and vGPUs, select them; otherwise, try to select them in the same NUMA node, the same CPU Socket, and across CPU Sockets in turn; Step 606: Kubelet obtains the specific information of the selected device from Device Plugin and allocates container resources.
[0089] It should be noted that, during the resource allocation process, the above steps 603, 604 and 605 can be performed simultaneously or separately.
[0090] In one embodiment, for a single-NUMA node host, where all one or more CPU sockets are logically considered to be the same NUMA node, the NUMA node layer in this solution can be skipped, and a PCIe topology structure consisting of a CPU socket layer and a PCIe switch layer can be constructed, as shown in Figure 7. During the allocation of CPU, memory, network card, and GPU resources, the search process for different NUMA nodes can be skipped based on the original technical solution. For example, if there are no resources to be allocated on the same PCIe switch, a match can be directly attempted on the same CPU socket, thereby completing the nearest allocation of container resources at the PCIe switch and CPU socket levels.
[0091] It should be noted that this method addresses the need for container orchestration systems like K8s to allocate host resources locally when constructing containers in RDMA virtualization scenarios. By reading the PCIe information of components like the CPU, network card, and GPU, and their mapping relationship to virtualized instances, it generates an intra-machine PCIe topology consisting of a hierarchical structure of CPU sockets, NUMA nodes, and PCIe switches. This topology enables local scheduling of CPUs, memory, VFs, and vGPUs, addressing the affinity issues of network and computing resources within containers. Compared to existing random scheduling, this solution can improve computing efficiency and reduce communication costs. Compared to affinity scheduling based solely on NUMA nodes, this solution can further achieve local resource allocation at the CPU socket and PCIe switch levels and is adaptable to single-NUMA node hosts, offering significant performance advantages in virtualized environments.
[0092] An embodiment of the present application provides a device for generating a host PCIe topology structure, which includes: a first acquisition module, used to respectively acquire a first topology structure and a second topology structure; wherein the first topology structure is used to represent a topology structure formed by a network card, a network card virtual function, a graphics processor, and a virtual graphics processor associated with the host under a PCIe adapter unit; the second topology structure is used to represent a topology structure formed by a CPU interface unit, a CPU core, and a memory based on the association between the host and a non-uniform memory access (NUMA) node; a second acquisition module, used to acquire an association relationship between the PCIe adapter unit and the NUMA node; and an association module, used to associate the first topology structure and the second topology structure based on the association relationship to obtain the PCIe topology structure of the host.
[0093] In an embodiment of the present application, the device for generating the host PCIe topology structure realizes the layer-by-layer proximity matching of the CPU, memory, network card, GPU and their virtualized instances by constructing a multi-layer topology structure of host resources from the CPU interface unit, NUMA node to the PCIe adapter unit, thereby solving the technical problem in the related technology that the container orchestration system such as K8s has difficulty in obtaining the topological relationship of the host computing, network and other components, and the allocated resources may have affinity allocation problems, resulting in unnecessary data transmission when the container performs computing tasks, resulting in increased latency and waste of resources. The technical effect is achieved that the affinity allocation problem of resources in the container can be solved to a large extent, unnecessary data transmission is avoided, and the computing performance of the container is improved.
[0094] It should be noted that the first acquisition module, the second acquisition module, and the association module described above correspond to steps S202 to S206 in the method embodiment. The examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in the method embodiment described above. It should be noted that the modules described above, as part of the apparatus, can be run in the electronic device provided in the embodiments of this application.
[0095] According to another aspect of an embodiment of the present application, a container resource allocation device is provided, the device comprising:
[0096] A third acquisition module is configured to obtain resources to be allocated to the container; and an allocation module is configured to allocate resources to the container based on the PCIe topology of the host when the allocatable resources of the host satisfy the resources to be allocated, wherein the PCIe topology of the host is obtained based on any of the above methods.
[0097] In an embodiment of the present application, the container resource allocation device realizes the layer-by-layer proximity matching of CPU, memory, network card, GPU and their virtualized instances by constructing a multi-layer topology structure of host resources from CPU interface unit, NUMA node to PCIe adapter unit, thereby solving the technical problem in related technologies that container orchestration systems such as K8s have difficulty in obtaining the topological relationship of host computing, network and other components, and their allocated resources may have affinity allocation problems, resulting in unnecessary data transmission when the container performs computing tasks, resulting in increased latency and waste of resources. The device achieves the technical effect of being able to solve the affinity allocation problem of resources in the container to a large extent, avoid unnecessary data transmission, and improve the computing performance of the container.
[0098] It should be noted that the third acquisition module and allocation module described above correspond to steps S502 to S504 in the method embodiment. The examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in the method embodiment described above. It should be noted that the modules described above, as part of the apparatus, can be run in the electronic device provided in the embodiments of the present application.
[0099] An embodiment of the present application further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform the method of the embodiment of the present application.
[0100] In one implementation, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network. The electronic device includes but is not limited to a computer device.
[0101] In one implementation, in this embodiment, the computer device may include: a memory and a processor, the memory storing a computer program; the processor, configured to execute the computer program stored in the memory, wherein when the computer program runs, the processor executes any one of the above method steps.
[0102] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for generating the host PCIe topology structure in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for generating the host PCIe topology structure. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.
[0103] In one embodiment, the processor may call information and applications stored in a memory through a transmission device to perform the following steps: obtaining a first topology structure and a second topology structure respectively; wherein the first topology structure is used to represent a topology structure formed by a network card, a network card virtual function, a graphics processor, and a virtual graphics processor associated with a host under a PCIe adapter unit; and the second topology structure is used to represent a topology structure formed based on a CPU interface unit, a CPU core, and a memory associated with a NUMA node of the host; obtaining an association relationship between the PCIe adapter unit and the NUMA node; and associating the first topology structure and the second topology structure based on the association relationship to obtain a PCIe topology structure of the host.
[0104] In one implementation, in this embodiment, the processor may further execute program code for the following steps: obtaining a first topology structure, including: obtaining a PCIe path of a network card, a NUMA node to which the network card belongs, a network card virtual function, and a mapping relationship between the network card and the network card virtual function; obtaining a PCIe path of a graphics processor, a NUMA node to which the graphics processor belongs, a virtual graphics processor, and a mapping relationship between the graphics processor and the virtual graphics processor; identifying the network card and the graphics processor under the same PCIe adapter unit based on the PCIe path of the network card and the PCIe path of the graphics processor; and obtaining the first topology structure based on the NUMA node to which the network card belongs, the network card virtual function, the mapping relationship between the network card and the network card virtual function, the NUMA node to which the graphics processor belongs, the virtual graphics processor, the mapping relationship between the graphics processor and the virtual graphics processor, and the association relationship between the network card and the graphics processor under the same PCIe adapter unit.
[0105] In one implementation, in this embodiment, the processor may further execute program code for the following steps: obtaining a first topology based on the NUMA node to which the network card belongs, the network card virtual function, the mapping relationship between the network card and the network card virtual function, the NUMA node to which the graphics processor belongs, the virtual graphics processor, the mapping relationship between the graphics processor and the virtual graphics processor, and the association relationship between the network card and the graphics processor under the same PCIe adapter unit, including: obtaining a first sub-topology based on the association relationship between the network card and the graphics processor under the same PCIe adapter unit; associating the network card virtual function with the first sub-topology based on the mapping relationship between the network card and the network card virtual function, and associating the NUMA node to which the network card belongs with the first sub-topology to obtain a first target topology; associating the virtual graphics processor with the first sub-topology based on the mapping relationship between the graphics processor and the virtual graphics processor, and associating the NUMA node to which the graphics processor belongs with the first sub-topology to obtain a second target topology; and associating the first target topology with the second target topology to obtain a first topology.
[0106] In one embodiment, in this embodiment, the above-mentioned processor can also execute the program code of the following steps: obtaining a second topology structure, including: obtaining a CPU interface unit, a NUMA node contained in the CPU interface unit, a CPU core, a NUMA node to which the CPU core belongs, a memory, and a NUMA node to which the memory belongs; obtaining the second topology structure based on the association relationship among the CPU interface unit, the NUMA node contained in the CPU interface unit, the CPU core, the NUMA node to which the CPU core belongs, the memory, and the NUMA node to which the memory belongs.
[0107] In one embodiment, in this embodiment, the above-mentioned processor can also execute the program code of the following steps: based on the association relationship between the CPU interface unit, the NUMA node contained in the CPU interface unit, the CPU core, the NUMA node to which the CPU core belongs, the memory, and the NUMA node to which the memory belongs, obtain a second topology structure, including: based on the association relationship between the CPU interface unit and the NUMA node contained in the CPU interface unit, obtain a second sub-topology structure; based on the NUMA node to which the CPU core belongs and the NUMA node to which the memory belongs, respectively associate the CPU core and the memory to the second sub-topology structure to obtain a second topology structure.
[0108] In addition, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the container resource allocation method and apparatus in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-mentioned container resource allocation method.
[0109] In one embodiment, the processor may call information and applications stored in the memory through a transmission device to perform the following steps: obtaining proposed allocated resources for the container; and, when allocable resources of the host satisfy the proposed allocated resources, allocating resources to the container based on a PCIe topology of the host, wherein the PCIe topology of the host is obtained based on any of the above methods.
[0110] In one embodiment, in this embodiment, the processor may further execute program code of the following steps: allocating resources to the container based on the PCIe topology of the host, including: determining a NUMA node whose remaining resources are greater than a preset remaining resource as a target NUMA node; and allocating allocable resources to the container based on the target NUMA node.
[0111] In one embodiment, in this embodiment, the above-mentioned processor may further execute program code of the following steps: when the allocatable resource is a CPU, allocatable resources are allocated to the container based on the target NUMA node, including: determining whether there are idle CPU cores in the target NUMA node, and whether the number of idle CPU cores meets the number of CPU cores required by the container; if there are idle CPU cores in the target NUMA node and the number of idle CPU cores meets the number of CPU cores required by the container, allocating the idle CPU cores to the container; if there are no idle CPU cores in the target NUMA node or the number of idle CPU cores does not meet the number of CPU cores required by the container, allocating the CPU cores under the same CPU interface unit as the target NUMA node to the container.
[0112] In one embodiment, in this embodiment, the processor may further execute program code of the following steps: allocating CPU cores under the same CPU interface unit as the target NUMA node to the container, including: determining whether the number of CPU cores under the same CPU interface unit meets the number of CPU cores required by the container; if the number of CPU cores under the same CPU interface unit meets the number of CPU cores required by the container, allocating the CPU cores under the same CPU interface unit to the container; if the number of CPU cores under the same CPU interface unit does not meet the number of CPU cores required by the container, allocating the CPU cores under a CPU interface unit different from the target NUMA node to the container.
[0113] In one embodiment, in this embodiment, the processor may further execute program code of the following steps: when the allocatable resource is memory, allocating the allocatable resource to the container based on the target NUMA node, including: determining whether the remaining memory of the target NUMA node satisfies the memory required by the container; if the remaining memory of the target NUMA node satisfies the memory required by the container, allocating the remaining memory of the target NUMA node to the container; if the remaining memory of the target NUMA node does not satisfy the memory required by the container, allocating the remaining memory of other NUMA nodes under the same CPU interface unit to the container.
[0114] In one embodiment, in this embodiment, the processor may further execute program code of the following steps: allocating the remaining memory of other NUMA nodes under the same CPU interface unit to the container, including: determining whether the remaining memory of other NUMA nodes under the same CPU interface unit satisfies the memory required by the container; if the remaining memory of other NUMA nodes satisfies the memory required by the container, allocating the remaining memory of other NUMA nodes to the container; if the remaining memory of other NUMA nodes does not satisfy the memory required by the container, allocating memory under a CPU interface unit different from the target NUMA node to the container.
[0115] In one implementation manner, in this embodiment, the processor may further execute program code for the following steps: when the allocatable resources are network card virtual functions and virtual graphics processors, before allocating the allocatable resources to the container based on the target NUMA node, the method may further include: determining whether the network card virtual functions and virtual graphics processors of the same PCIe adapter unit meet the network card virtual functions and virtual graphics processors required by the container; if the network card virtual functions and virtual graphics processors of the same PCIe adapter unit meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors of the same PCIe adapter unit to the container.
[0116] In one implementation manner, in this embodiment, the processor may further execute program code for the following steps: when the allocatable resources are network card virtual functions and virtual graphics processors, allocating the allocatable resources to the container based on the target NUMA node, including: determining whether the network card virtual functions and virtual graphics processors of the target NUMA node meet the network card virtual functions and virtual graphics processors required by the container; if the network card virtual functions and virtual graphics processors of the target NUMA node meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors of the target NUMA node to the container; if the network card virtual functions and virtual graphics processors of the target NUMA node do not meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors under the same CPU interface unit as the target NUMA node to the container.
[0117] In one implementation manner, in this embodiment, the processor may further execute program code for the following steps: allocating a network card virtual function and a virtual graphics processor in the same CPU interface unit as the target NUMA node to the container, including: determining whether the network card virtual function and the virtual graphics processor in the same CPU interface unit as the target NUMA node meet the network card virtual function and the virtual graphics processor required by the container; if the network card virtual function and the virtual graphics processor in the same CPU interface unit as the target NUMA node meet the network card virtual function and the virtual graphics processor required by the container, allocating the network card virtual function and the virtual graphics processor in the same CPU interface unit to the container; if the network card virtual function and the virtual graphics processor in the same CPU interface unit as the target NUMA node do not meet the network card virtual function and the virtual graphics processor required by the container, allocating the network card virtual function and the virtual graphics processor in a different CPU interface unit to the container.
[0118] In one embodiment, in this embodiment, the above-mentioned processor can also execute the program code of the following steps: determining a NUMA node whose remaining resources are greater than the preset remaining resources as a target NUMA node, including: obtaining the remaining resources of the NUMA node and the weight coefficient corresponding to the remaining resources; calculating the remaining resource evaluation value of the NUMA node based on the remaining resources and the weight coefficient corresponding to the remaining resources; obtaining the preset remaining resource evaluation value corresponding to the preset remaining resources; when the remaining resource evaluation value is greater than the preset remaining resource evaluation value, determining that the remaining resources of the NUMA node are greater than the preset remaining resources, and taking the NUMA node as the target NUMA node.
[0119] In one embodiment, in this embodiment, the processor may further execute the program code of the following steps: the method further includes: if there are multiple NUMA nodes whose remaining resources are greater than the preset remaining resources, the NUMA node with the largest remaining resource evaluation value is used as the target NUMA node.
[0120] An embodiment of the present application further provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.
[0121] In one implementation manner, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.
[0122] In one embodiment, in this embodiment, the above-mentioned storage medium can be used to store the program code corresponding to the host PCIe topology structure generation method provided in the above-mentioned embodiment, and when the program code is run by the processor, the processor is controlled to execute the above-mentioned host PCIe topology structure generation method.
[0123] In one embodiment, in this example, the storage medium is configured to store program code for executing the following steps: obtaining a first topology structure and a second topology structure respectively; wherein the first topology structure is used to represent a topology structure formed by the network card, network card virtual function, graphics processor, and virtual graphics processor associated with the host under the PCIe adapter unit; the second topology structure is used to represent a topology structure formed by the CPU interface unit, CPU core, and memory associated with the host and the NUMA node; obtaining an association relationship between the PCIe adapter unit and the NUMA node; and associating the first topology structure and the second topology structure based on the association relationship to obtain the PCIe topology structure of the host.
[0124] In one embodiment, in this example, the storage medium is configured to store program code further for executing the following steps: obtaining a first topology structure, including: obtaining a PCIe path of a network card, a NUMA node to which the network card belongs, a network card virtual function, and a mapping relationship between the network card and the network card virtual function; obtaining a PCIe path of a graphics processor, a NUMA node to which the graphics processor belongs, a virtual graphics processor, and a mapping relationship between the graphics processor and the virtual graphics processor; identifying the network card and the graphics processor under the same PCIe adapter unit based on the PCIe path of the network card and the PCIe path of the graphics processor; obtaining the first topology structure based on the NUMA node to which the network card belongs, the network card virtual function, the mapping relationship between the network card and the network card virtual function, the NUMA node to which the graphics processor belongs, the virtual graphics processor, the mapping relationship between the graphics processor and the virtual graphics processor, and the association relationship between the network card and the graphics processor under the same PCIe adapter unit.
[0125] In one embodiment, in this embodiment, the storage medium is configured to store program code further for executing the following steps: obtaining a first topology structure based on the NUMA node to which the network card belongs, the network card virtual function, the mapping relationship between the network card and the network card virtual function, the NUMA node to which the graphics processor belongs, the virtual graphics processor, the mapping relationship between the graphics processor and the virtual graphics processor, and the association relationship between the network card and the graphics processor under the same PCIe adapter unit, including: obtaining a first sub-topology structure based on the association relationship between the network card and the graphics processor under the same PCIe adapter unit; associating the network card virtual function with the first sub-topology structure based on the mapping relationship between the network card and the network card virtual function, and associating the NUMA node to which the network card belongs with the first sub-topology structure, to obtain a first target topology structure; associating the virtual graphics processor with the first sub-topology structure based on the mapping relationship between the graphics processor and the virtual graphics processor, and associating the NUMA node to which the graphics processor belongs with the first sub-topology structure, to obtain a second target topology structure; associating the first target topology structure with the second target topology structure to obtain a first topology structure.
[0126] In one embodiment, in this embodiment, the above-mentioned storage medium is configured to store program codes further for executing the following steps: obtaining a second topology structure, including: obtaining a CPU interface unit, a NUMA node contained in the CPU interface unit, a CPU core, a NUMA node to which the CPU core belongs, a memory, and a NUMA node to which the memory belongs; obtaining the second topology structure based on the association relationship among the CPU interface unit, the NUMA node contained in the CPU interface unit, the CPU core, the NUMA node to which the CPU core belongs, the memory, and the NUMA node to which the memory belongs.
[0127] In one embodiment, in this embodiment, the above-mentioned storage medium is configured to store program code for executing the following steps: obtaining a second topology structure based on the association relationship between the CPU interface unit, the NUMA node contained in the CPU interface unit, the CPU core, the NUMA node to which the CPU core belongs, the memory, and the NUMA node to which the memory belongs, including: obtaining a second sub-topology structure based on the association relationship between the CPU interface unit and the NUMA node contained in the CPU interface unit; associating the CPU core and the memory to the second sub-topology structure based on the NUMA node to which the CPU core belongs and the NUMA node to which the memory belongs, to obtain a second topology structure.
[0128] In one implementation, in this embodiment, the storage medium may be used to store program code corresponding to the container resource allocation method provided in the above embodiment, and control the processor to execute the container resource allocation method when the program code is executed by the processor.
[0129] In one embodiment, in this example, the storage medium is configured to store program code for executing the following steps: obtaining proposed allocated resources for the container; and allocating resources to the container based on a PCIe topology of the host when allocatable resources of the host satisfy the proposed allocated resources, wherein the PCIe topology of the host is obtained based on any of the above methods.
[0130] In one embodiment, in this embodiment, the above-mentioned storage medium is configured to store program code for executing the following steps: allocating resources to the container based on the PCIe topology structure of the host, including: determining a NUMA node whose remaining resources are greater than a preset remaining resource as a target NUMA node; and allocating allocatable resources to the container based on the target NUMA node.
[0131] In one embodiment, in this embodiment, the above-mentioned storage medium is configured to store program codes further for executing the following steps: when the allocatable resource is a CPU, allocating the allocatable resource to the container based on the target NUMA node, including: determining whether there are idle CPU cores in the target NUMA node, and whether the number of idle CPU cores meets the number of CPU cores required by the container; if there are idle CPU cores in the target NUMA node and the number of idle CPU cores meets the number of CPU cores required by the container, allocating the idle CPU cores to the container; if there are no idle CPU cores in the target NUMA node or the number of idle CPU cores does not meet the number of CPU cores required by the container, allocating the CPU cores under the same CPU interface unit as the target NUMA node to the container.
[0132] In one embodiment, in this embodiment, the above-mentioned storage medium is configured to store program codes further for executing the following steps: allocating CPU cores under the same CPU interface unit as the target NUMA node to the container, including: determining whether the number of CPU cores under the same CPU interface unit meets the number of CPU cores required by the container; if the number of CPU cores under the same CPU interface unit meets the number of CPU cores required by the container, allocating the CPU cores under the same CPU interface unit to the container; if the number of CPU cores under the same CPU interface unit does not meet the number of CPU cores required by the container, allocating the CPU cores under a CPU interface unit different from the target NUMA node to the container.
[0133] In one embodiment, in this embodiment, the above-mentioned storage medium is configured to store program codes further for executing the following steps: when the allocatable resource is memory, allocating the allocatable resource to the container based on the target NUMA node, including: determining whether the remaining memory of the target NUMA node meets the memory required by the container; if the remaining memory of the target NUMA node meets the memory required by the container, allocating the remaining memory of the target NUMA node to the container; if the remaining memory of the target NUMA node does not meet the memory required by the container, allocating the remaining memory of other NUMA nodes under the same CPU interface unit to the container.
[0134] In one embodiment, in this embodiment, the above-mentioned storage medium is configured to store program codes further for executing the following steps: allocating the remaining memory of other NUMA nodes under the same CPU interface unit to the container, including: determining whether the remaining memory of other NUMA nodes under the same CPU interface unit meets the memory required by the container; if the remaining memory of other NUMA nodes meets the memory required by the container, allocating the remaining memory of other NUMA nodes to the container; if the remaining memory of other NUMA nodes does not meet the memory required by the container, allocating memory under a CPU interface unit different from the target NUMA node to the container.
[0135] In one embodiment, in this example, the storage medium is configured to store program code further for executing the following steps: when the allocatable resources are network card virtual functions and virtual graphics processors, before allocating the allocatable resources to the container based on the target NUMA node, the method further includes: determining whether the network card virtual functions and virtual graphics processors of the same PCIe adapter unit meet the network card virtual functions and virtual graphics processors required by the container; if the network card virtual functions and virtual graphics processors of the same PCIe adapter unit meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors of the same PCIe adapter unit to the container.
[0136] In one embodiment, in this example, the storage medium is configured to store program code further for executing the following steps: when the allocatable resources are network card virtual functions and virtual graphics processors, allocating the allocatable resources to the container based on the target NUMA node, including: determining whether the network card virtual functions and virtual graphics processors of the target NUMA node meet the network card virtual functions and virtual graphics processors required by the container; if the network card virtual functions and virtual graphics processors of the target NUMA node meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors of the target NUMA node to the container; if the network card virtual functions and virtual graphics processors of the target NUMA node do not meet the network card virtual functions and virtual graphics processors required by the container, allocating the network card virtual functions and virtual graphics processors under the same CPU interface unit as the target NUMA node to the container.
[0137] In one embodiment, in this example, the storage medium is configured to store program code further for executing the following steps: allocating a network card virtual function and a virtual graphics processor in the same CPU interface unit as a target NUMA node to a container, including: determining whether the network card virtual function and the virtual graphics processor in the same CPU interface unit as the target NUMA node meet the network card virtual function and the virtual graphics processor required by the container; if the network card virtual function and the virtual graphics processor in the same CPU interface unit as the target NUMA node meet the network card virtual function and the virtual graphics processor required by the container, allocating the network card virtual function and the virtual graphics processor in the same CPU interface unit to the container; if the network card virtual function and the virtual graphics processor in the same CPU interface unit as the target NUMA node do not meet the network card virtual function and the virtual graphics processor required by the container, allocating the network card virtual function and the virtual graphics processor in a different CPU interface unit to the container.
[0138] In one embodiment, in this embodiment, the above-mentioned storage medium is configured to store program codes for executing the following steps: determining a NUMA node whose remaining resources are greater than preset remaining resources as a target NUMA node, including: obtaining the remaining resources of the NUMA node and the weight coefficient corresponding to the remaining resources; calculating the remaining resource evaluation value of the NUMA node based on the remaining resources and the weight coefficient corresponding to the remaining resources; obtaining a preset remaining resource evaluation value corresponding to the preset remaining resources; when the remaining resource evaluation value is greater than the preset remaining resource evaluation value, determining that the remaining resources of the NUMA node are greater than the preset remaining resources, and using the NUMA node as the target NUMA node.
[0139] In one embodiment, in this embodiment, the above-mentioned storage medium is configured to store program code for executing the following steps: the above-mentioned method also includes: if there are multiple NUMA nodes whose remaining resources are greater than the preset remaining resources, then the NUMA node with the largest remaining resource evaluation value is used as the target NUMA node.
[0140] In an embodiment of the present application, a first topology structure and a second topology structure are obtained respectively; wherein the first topology structure is used to represent the topology structure formed by the network card, network card virtual function, graphics processor and virtual graphics processor associated with the host under the PCIe adapter unit; the second topology structure is used to represent the topology structure formed by the CPU interface unit, CPU core and memory associated with the host and the NUMA node; the association relationship between the PCIe adapter unit and the NUMA node is obtained; and the first topology structure and the second topology structure are associated based on the association relationship to obtain the PCIe topology structure of the host. In other words, the embodiment of the present application achieves layer-by-layer proximity matching of CPU, memory, network card, GPU and their virtualized instances by constructing a multi-layer topology structure of host resources from CPU interface unit, NUMA node to PCIe adapter unit, thereby solving the technical problem in the related art that the container orchestration system such as K8s has difficulty in obtaining the topological relationship of the host computing, network and other components, and the resources allocated thereto may have affinity allocation problems, resulting in unnecessary data transmission when the container executes the computing task, resulting in increased latency and resource waste, thereby achieving the technical effect of being able to largely solve the affinity allocation problem of resources in the container, avoid unnecessary data transmission, and improve the computing performance of the container.
[0141] An embodiment of the present application further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.
[0142] With reference to Figure 8, the structural block diagram of the electronic device that can be used as the server or client of the embodiment of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0143] As shown in Figure 8, the electronic device includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device can also be stored. The computing unit 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0144] Multiple components within the electronic device are connected to the I / O interface 805, including an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device capable of inputting information into the electronic device. The input unit 806 can receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 807 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 808 may include, but is not limited to, a magnetic disk or an optical disk. The communication unit 809 allows the electronic device to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0145] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units for running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present application may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 802 and / or communication unit 809. In some embodiments, the computing unit 801 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).
[0146] The computer programs for implementing the methods of the embodiments of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0147] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable signal medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0148] It should be noted that the term "including" and its variations used in the embodiments of the present application are open inclusions, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present application are illustrative and not restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".
[0149] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0150] The various steps described in the method implementation schemes provided in the embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method implementation schemes may include additional steps and / or omit the steps shown. The scope of protection of the present application is not limited in this respect.
[0151] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments are referenced to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts are referred to the partial description of the method embodiment.
[0152] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.
Claims
1. A method for generating a host PCIe topology structure, comprising: Acquire a first topology structure and a second topology structure respectively; wherein the first topology structure is used to characterize a topology structure formed by a network card, a network card virtual function, a graphics processor, and a virtual graphics processor associated with a host under a PCIe adapter unit; and the second topology structure is used to characterize a topology structure formed by a CPU interface unit, a CPU core, and a memory associated with a non-uniform memory access NUMA node based on the host; Obtaining an association relationship between the PCIe adapter unit and the NUMA node; The first topology structure and the second topology structure are associated with each other based on the association relationship to obtain a PCIe topology structure of the host.
2. The method according to claim 1, wherein: Obtaining a first topology structure, including: Obtaining a PCIe path of the network card, a NUMA node to which the network card belongs, the network card virtual function, and a mapping relationship between the network card and the network card virtual function; Obtaining a PCIe path of the graphics processor, a NUMA node to which the graphics processor belongs, the virtual graphics processor, and a mapping relationship between the graphics processor and the virtual graphics processor; Based on the PCIe path of the network card and the PCIe path of the graphics processor, identifying the network card and the graphics processor under the same PCIe adapter unit; The first topology structure is obtained based on the NUMA node to which the network card belongs, the network card virtual function, and a mapping relationship between the network card and the network card virtual function, the NUMA node to which the graphics processor belongs, the virtual graphics processor, and a mapping relationship between the graphics processor and the virtual graphics processor, and an association relationship between the network card and the graphics processor under the same PCIe adapter unit.
3. The method according to claim 2, wherein: Based on the NUMA node to which the network card belongs, the network card virtual function, the mapping relationship between the network card and the network card virtual function, the NUMA node to which the graphics processor belongs, the virtual graphics processor, the mapping relationship between the graphics processor and the virtual graphics processor, and the association relationship between the network card and the graphics processor under the same PCIe adapter unit, acquiring the first topology structure includes: Based on the association relationship between the network card and the graphics processor under the same PCIe adapter unit, a first sub-topology structure is obtained; Associating the network card virtual function with the first sub-topology structure based on the mapping relationship between the network card and the network card virtual function, and associating the NUMA node to which the network card belongs with the first sub-topology structure to obtain a first target topology structure; Associating the virtual graphics processor with the first sub-topology structure based on the mapping relationship between the graphics processor and the virtual graphics processor, and associating the NUMA node to which the graphics processor belongs with the first sub-topology structure to obtain a second target topology structure; The first target topological structure and the second target topological structure are associated to obtain the first topological structure.
4. The method according to claim 1, wherein: The second topology structure is obtained, including: Obtain the CPU interface unit, the NUMA node included in the CPU interface unit, the CPU core, the NUMA node to which the CPU core belongs, the memory, and the NUMA node to which the memory belongs; The second topology structure is acquired based on the association relationship among the CPU interface unit, the NUMA node included in the CPU interface unit, the CPU core, the NUMA node to which the CPU core belongs, the memory, and the NUMA node to which the memory belongs.
5. The method according to claim 4, wherein: Acquiring the second topology structure based on the association relationship between the CPU interface unit, the NUMA node included in the CPU interface unit, the CPU core, the NUMA node to which the CPU core belongs, the memory, and the NUMA node to which the memory belongs includes: Acquire a second sub-topology structure based on an association relationship between the CPU interface unit and a NUMA node included in the CPU interface unit; Based on the NUMA node to which the CPU core belongs and the NUMA node to which the memory belongs, the CPU core and the memory are respectively associated with the second sub-topology structure to obtain the second topology structure.
6. A container resource allocation method, comprising: Get the resources to be allocated to the container; When the allocatable resources of the host satisfy the resources to be allocated, resources are allocated to the container based on the PCIe topology structure of the host, wherein the PCIe topology structure of the host is obtained based on the method of any one of claims 1 to 5.
7. The method according to claim 6, wherein: Allocating resources to the container based on the PCIe topology of the host includes: Determine a NUMA node whose remaining resources are greater than a preset remaining resource as a target NUMA node; The allocatable resources are allocated to the containers based on the target NUMA node.
8. The method according to claim 7, wherein: When the allocatable resource is a CPU, allocating the allocatable resource to the container based on the target NUMA node includes: Determine whether there are idle CPU cores on the target NUMA node, and whether the number of the idle CPU cores meets the number of CPU cores required by the container; If the target NUMA node has idle CPU cores and the number of the idle CPU cores meets the number of CPU cores required by the container, the idle CPU cores are allocated to the container; If the target NUMA node does not have any idle CPU cores or the number of the idle CPU cores does not meet the number of CPU cores required by the container, a CPU core under the same CPU interface unit as the target NUMA node is allocated to the container.
9. The method according to claim 8, wherein: Allocating a CPU core in the same CPU interface unit as the target NUMA node to the container includes: Determine whether the number of CPU cores under the same CPU interface unit meets the number of CPU cores required by the container; If the number of CPU cores under the same CPU interface unit meets the number of CPU cores required by the container, the CPU cores under the same CPU interface unit are allocated to the container; If the number of CPU cores under the same CPU interface unit does not meet the number of CPU cores required by the container, a CPU core under a CPU interface unit different from the target NUMA node is allocated to the container.
10. The method according to claim 7, wherein: When the allocatable resource is memory, allocating the allocatable resource to the container based on the target NUMA node includes: Determine whether the remaining memory of the target NUMA node meets the memory required by the container; If the remaining memory of the target NUMA node satisfies the memory required by the container, the remaining memory of the target NUMA node is allocated to the container; If the remaining memory of the target NUMA node does not meet the memory required by the container, the remaining memory of other NUMA nodes under the same CPU interface unit is allocated to the container.
11. The method according to claim 10, wherein: Allocate the remaining memory of other NUMA nodes under the same CPU interface unit to the container, including: Determine whether the remaining memory of other NUMA nodes under the same CPU interface unit meets the memory required by the container; If the remaining memory of the other NUMA nodes meets the memory required by the container, allocating the remaining memory of the other NUMA nodes to the container; If the remaining memory of the other NUMA nodes does not meet the memory required by the container, the memory under the CPU interface unit different from the target NUMA node is allocated to the container.
12. The method according to claim 7, wherein: When the allocatable resources are network card virtual functions and virtual graphics processors, before allocating the allocatable resources to the container based on the target NUMA node, the method further includes: Determine whether the network card virtual function and the virtual graphics processor of the same PCIe adapter unit meet the network card virtual function and the virtual graphics processor required by the container; If the network card virtual function and the virtual graphics processor of the same PCIe adapter unit meet the network card virtual function and the virtual graphics processor required by the container, the network card virtual function and the virtual graphics processor of the same PCIe adapter unit are allocated to the container.
13. The method according to claim 7, wherein: When the allocatable resources are network card virtual functions and virtual graphics processors, allocating the allocatable resources to the container based on the target NUMA node includes: Determine whether the network card virtual function and the virtual graphics processor of the target NUMA node meet the network card virtual function and the virtual graphics processor required by the container; If the network card virtual function and the virtual graphics processor of the target NUMA node meet the network card virtual function and the virtual graphics processor required by the container, the network card virtual function and the virtual graphics processor of the target NUMA node are allocated to the container; If the network card virtual function and the virtual graphics processor of the target NUMA node do not meet the network card virtual function and the virtual graphics processor required by the container, the network card virtual function and the virtual graphics processor under the same CPU interface unit as the target NUMA node are allocated to the container.
14. The method according to claim 13, wherein: Allocating a network card virtual function and a virtual graphics processor in the same CPU interface unit as the target NUMA node to the container includes: Determine whether the network card virtual function and the virtual graphics processor in the same CPU interface unit as the target NUMA node meet the network card virtual function and the virtual graphics processor required by the container; If the network card virtual function and virtual graphics processor in the same CPU interface unit as the target NUMA node meet the network card virtual function and virtual graphics processor required by the container, the network card virtual function and virtual graphics processor in the same CPU interface unit are The network card virtual function and the virtual graphics processor are allocated to the container; If the network card virtual functions and virtual graphics processors under the same CPU interface unit as the target NUMA node do not meet the network card virtual functions and virtual graphics processors required by the container, the network card virtual functions and virtual graphics processors under different CPU interface units are allocated to the container.
15. An electronic device, comprising: Memory and processor; The memory is used to store a computer program, and the processor is used to execute the computer program. When the computer program is executed by the processor, the method according to any one of claims 1 to 14 is implemented.
16. A computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 14 when executed by a processor.
Citation Information
Patent Citations
NUMA-based network optimization method and system for resource global affinity under virtualized environment
CN107967180A
Container scheduling method and device, cluster system, electronic equipment and storage medium
CN114610497A
Electronic equipment and PCIe topology configuration method and device thereof
CN115905094A
Virtual machine NUMA topology generation method and device, equipment and medium
CN116069447A
Generation method of host PCIe topological structure and container resource allocation method
CN116755829A
Cited By
Container system process life cycle management method and system
CN119473636A
Access control method and device and server
CN119903010A
Task scheduling method and device, electronic equipment, medium and computer program product
CN120849064A
Resource allocation system and method applied to GPU cluster, electronic equipment, storage medium and computer program product
CN121187784A
Continuous integration task scheduling method and device
CN122570129A
