Ollama inference optimization method for domestic platform
By analyzing NUMA topology information on a domestic platform, selecting the optimal NUMA node combination and binding it to the ollama runner process, the problem of CPU cross-node memory access was solved, inference performance was improved, and a 60% performance improvement was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- KYLIN CORP
- Filing Date
- 2026-02-24
- Publication Date
- 2026-07-07
AI Technical Summary
Existing technologies on domestic platforms do not take into account the characteristics of NUMA architecture, which leads to a decrease in CPU memory access performance across NUMA nodes during the first stage of inference, affecting the overall inference throughput performance.
By acquiring the NUMA topology information of the domestic platform, the optimal combination of NUMA nodes is selected, and the ollama runner process is bound to the optimal NUMA node and its CPU to avoid cross-node memory access and improve memory access efficiency.
It improves the performance of the CPU inference stage and increases the overall inference throughput. In specific tests, the deep model inference performance on Kunpeng 920+Tesla T4 was improved by 60%.
Smart Images
Figure CN121706995B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an inference optimization method, and more particularly to an Ollama inference optimization method for domestic platforms, belonging to the field of artificial intelligence technology. Background Technology
[0002] like Figure 3 The diagram shown illustrates the overall architecture of the Ollama inference framework. Ollama employs a classic Client-Server (CS) architecture. Users interact with the Ollama HTTP Server on the server side via the Ollama CLI command-line interface (e.g., `ollama run`). The Ollama HTTP Server can be started directly via the command line (e.g., `ollama serve`). Communication between the client and server is via HTTP.
[0003] Ollama Server has two core components:
[0004] ollama-http-server: Responsible for interacting with the client, and also starts an HTTP service through ollama runner for interaction with llama.cpp.
[0005] llama.cpp: As the LLM inference engine, it is responsible for loading and running large language models, processing inference requests and returning results. It is an independent open source project with cross-platform and hardware friendliness, and can run in environments without GPUs, i.e., only CPUs.
[0006] ollama-http-server and llama.cpp also interact via HTTP.
[0007] Taking ollama running deepseek-r1:32b as an example, this section introduces the inference process of ollama running large models:
[0008] 1. First, start the ollama server in the terminal: ollama serve.
[0009] 2. Next, open a new terminal and run deepseek-r1:32b: ollama run deepseek-r1:32b; enter the dialog terminal, and you can now start communicating with the deepseek-r1:32b large model.
[0010] In the aforementioned existing technology, when a user runs `ollama run deepseek-r1:32b` and enters the dialog terminal, `ollama-http-server` executes `ollama runner` to start the HTTP service for model inference. When the user begins to interact with the `deepseek-r1:32b` model in the dialog terminal, `ollama-http-server` calls `llama.cpp` through the HTTP service started by `ollama runner` to perform inference and returns the dialogue result to the user. `llama.cpp` is a heterogeneous computing framework, meaning it supports heterogeneous inference using both GPUs and CPUs. During the inference process, `llama.cpp` relies on the CPU for the first stage of inference, then passes the inference result to the GPU for subsequent stages of inference, and finally returns the inference result to the user. The current ollama inference method does not consider the NUMA architecture characteristics of domestic platforms. During the first stage of CPU inference, the CPU may access memory across NUMA nodes, leading to a decrease in CPU inference performance in the first stage and affecting the overall ollama inference throughput performance. Summary of the Invention
[0011] To address the aforementioned issues, this invention provides an ollama inference optimization method for domestic platforms, which reduces memory access across NUMA nodes performed by llama.cpp during the CPU inference phase.
[0012] To achieve the above objectives, the technical solution of the present invention is: an Ollama inference optimization method for domestic platforms, the method comprising the following steps:
[0013] S1. Start the Ollama HTTP Server using the ollama serve command;
[0014] S2. Send an inference request to the Ollam HTTP Server using the ollama run command;
[0015] S3, Ollam HTTP Server starts the Ollam runner process;
[0016] S4. Obtain the NUMA topology information of the node where ollama is located;
[0017] S5. Analyze the NUMA topology information of the nodes where ollama is located and select the optimal NUMA node combination;
[0018] S6. Bind the ollama runner process to the optimal NUMA node combination and the CPU where the optimal NUMA node combination is located;
[0019] S7, the ollama runner process calls llama.cpp to begin CPU-stage inference;
[0020] S8 and llama.cpp begin the GPU-stage inference;
[0021] S9 and llama.cpp return the inference results to the ollama runner process;
[0022] S10, ollama runner returns the final inference result to the user terminal that executed ollama run.
[0023] Furthermore, step S4 specifically includes the following steps:
[0024] S41. Obtain the NUMA node information of the node where ollama is located from the / sys / devices / system / node / nodeX file, where X is the code number 0, 1, 2, 3 and so on;
[0025] S42. Obtain the CPU information on the NUMA node based on / sys / devices / system / node / nodeX / CPUX of the node where ollama is located, where X is the code number 0, 1, 2, 3 and so on;
[0026] S43. Obtain the NUMA distance between nodes based on the / sys / devices / system / node / nodeX / distance file of the node where ollama is located, where X is the code number 0, 1, 2, 3 and so on.
[0027] Furthermore, step S5 specifically includes the following steps: Based on the NUMA topology information of the node where ollama is located obtained in step S4, and combined with the OLLAMA_NUM_THREADS thread data set by the user, select the optimal NUMA node combination.
[0028] Furthermore, if the OLLAMA_NUM_THREADS thread data is less than or equal to the number of CPUs on a single NUMA node, then a single NUMA node is selected as the optimal NUMA node combination.
[0029] Furthermore, if the number of threads in OLLAMA_NUM_THREADS is greater than the number of CPUs on a single NUMA node, then the NUMA node with the shortest NUMA distance is selected as the optimal NUMA node combination.
[0030] Furthermore, step S6 specifically includes the following steps: Based on the optimal NUMA node combination obtained in step S5, bind the ollama runner process to the optimal NUMA node combination and the CPU where the optimal NUMA node combination is located.
[0031] Furthermore, the ollamarunner process is bound to the optimal NUMA node combination by calling the set_mempolicy system function.
[0032] Furthermore, based on the optimal NUMA node combination, the sched_setaffinity system function is called to bind the ollamarunner process to the CPU where the optimal NUMA node combination resides.
[0033] The beneficial effects of the Ollama inference optimization method for domestic platforms proposed in this invention are as follows:
[0034] This invention, starting from the NUMA architecture of a domestic platform, designs a NUMA scheduling optimization algorithm. This algorithm analyzes the NUMA topology information of nodes, selects NUMA nodes with optimal distances, and automatically configures the optimal NUMA affinity parameters for large model inference. This binds the ollama runner process to these NUMA nodes and the CPUs where these NUMA nodes reside, preventing llama.cpp from accessing memory across NUMA nodes during the CPU inference phase. This improves memory access efficiency and the performance of llama.cpp during the CPU inference phase, thereby enhancing the throughput performance of mainstream large models in ollama inference. After adopting this invention's technical solution, actual testing showed that the inference performance of ollama running the deepseek-r1:32b model on a Kunpeng 920+Tesla T4 was improved by over 60%. Attached Figure Description
[0035] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0036] Figure 1 This is an overall design diagram of an Ollama inference optimization method for domestic platforms according to the present invention;
[0037] Figure 2 These are specific embodiments of the present invention.
[0038] Figure 3 This is a schematic diagram of the Ollama architecture. Detailed Implementation
[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0040] Definitions of abbreviations and key terms:
[0041] ollama is an open-source inference framework primarily used to run and manage large language models (LLMs) on local devices without relying on cloud services or the internet, thus ensuring data privacy and offline availability.
[0042] NUMA: Non-uniform memory access is a memory architecture designed for multiprocessor computers. It separates CPU resources and groups them into nodes. Each NUMA node has its own CPU, memory, and other resources. When resources within a NUMA node interact, the performance will be greatly improved. However, the performance of resource interaction between multiple NUMA nodes will be worse.
[0043] Domestic platforms: Servers using ARM64 architecture processors (such as Phytium, Huawei Kunpeng, etc.) and the Galaxy Kylin operating system.
[0044] Example 1
[0045] Combination Figure 1 As shown, this invention provides a detailed description of an Ollam inference optimization method for domestic platforms. The Ollam inference framework primarily comprises a client and a server, a classic client-server architecture. Users interact with the Ollam HTTP Server on the server side via the Ollam CLI command line (e.g., `ollama run`). The Ollam HTTP Server can be started directly via the command line (`ollama serve`). Communication between the client and server is via HTTP.
[0046] The server-side has two core components:
[0047] ollama-http-server: Responsible for interacting with the client, and also starts an HTTP service through ollama runner for interaction with llama.cpp.
[0048] llama.cpp: As the LLM inference engine, it is responsible for loading and running large language models, processing inference requests and returning results. It is an independent open source project with cross-platform and hardware friendliness, and can run in environments without GPUs, i.e., only CPUs.
[0049] ollama-http-server and llama.cpp also interact via HTTP.
[0050] The method of the present invention includes the following steps:
[0051] S1. Start the Ollama HTTP Server using the ollama serve command;
[0052] S2. Send an inference request to the Ollam HTTP Server using the ollama run command;
[0053] S3, Ollam HTTP Server starts the Ollam runner process;
[0054] S4. Obtain the NUMA topology information of the node where ollama is located;
[0055] S5. Analyze the NUMA topology information of the nodes where ollama is located and select the optimal NUMA node combination;
[0056] S6. Bind the ollama runner process to the optimal NUMA node combination and the CPU where the optimal NUMA node combination is located;
[0057] S7, the ollama runner process calls llama.cpp to begin CPU-stage inference;
[0058] S8 and llama.cpp begin the GPU-stage inference;
[0059] S9 and llama.cpp return the inference results to the ollama runner process;
[0060] S10, ollama runner returns the final inference result to the user terminal that executed ollama run.
[0061] Example 2
[0062] Based on Example 1, further combining Figure 1 Step S4 will be described in detail, including the following steps:
[0063] S41. Obtain the NUMA node information of the node where ollama is located from the / sys / devices / system / node / nodeX file, where X is a code number 0, 1, 2, 3 and so on;
[0064] S42. Obtain CPU information on the NUMA node based on / sys / devices / system / node / nodeX / CPUX of the node where ollama is located, where X is a code number 0, 1, 2, 3, etc.
[0065] S43. Obtain the NUMA distance between nodes based on the / sys / devices / system / node / nodeX / distance file of the node where ollama is located, where X is a code number 0, 1, 2, 3, etc.
[0066] Example 3
[0067] Based on Example 2, further combining Figure 1 Step S5 will be described in detail, including the following steps:
[0068] S51. Based on the NUMA topology information of the node where ollama is located obtained in step S4, and combined with the OLLAMA_NUM_THREADS thread data set by the user, select the optimal NUMA node combination. There are two specific cases.
[0069] S52. If the OLLAMA_NUM_THREADS thread data is less than or equal to the number of CPUs on a single NUMA node, then select a single NUMA node as the optimal NUMA node combination.
[0070] S53. If the number of threads in OLLAMA_NUM_THREADS is greater than the number of CPUs on a single NUMA node, then the NUMA node with the shortest NUMA distance is selected as the optimal NUMA node combination.
[0071] Example 4
[0072] Based on Example 3, further combining Figure 1 Step S6 will be described in detail, including the following steps:
[0073] S61. Perform binding operations based on the optimal NUMA node combination obtained in step S5;
[0074] S62. Bind the ollama runner process to the optimal NUMA node combination by calling the set_mempolicy system function;
[0075] S63. Based on the optimal NUMA node combination, call the sched_setaffinity system function to bind the ollama runner process to the CPU where the optimal NUMA node combination is located.
[0076] Example 5
[0077] Based on the above embodiments, combined with Figure 2 The specific implementation of the method of the present invention is described below, which specifically includes the following steps:
[0078] Step 101: Execute the ollama serve command to start the Ollama HTTP Server service;
[0079] Step 102: The user executes the ollama run command to run the large model;
[0080] Step 103: Select the optimal NUMA node combination;
[0081] Step 104: Bind the ollama runner process to the optimal NUMA node and its corresponding CPU;
[0082] Step 105, llama.cpp begins inference;
[0083] Step 106: Return the inference results to the user.
[0084] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
Claims
1. An Ollama inference optimization method for domestic platforms, characterized in that, The method includes the following steps: S1. Start the Ollama HTTP Server using the ollama serve command; S2. Send an inference request to the Ollam HTTP Server using the ollama run command; S3, Ollam HTTP Server starts the Ollam runner process; S4. Obtain the NUMA topology information of the node where ollama is located, including NUMA node information, CPU information on the NUMA node, and NUMA distance between NUMA nodes; S5. Analyze the NUMA topology information of the node where ollama is located and select the optimal NUMA node combination based on the user-set OLLAMA_NUM_THREADS thread data; if the thread data is less than or equal to the number of CPUs on a single NUMA node, then select a single NUMA node as the optimal combination; if the thread data is greater than the number of CPUs on a single NUMA node, then select the NUMA node with the shortest NUMA distance as the optimal combination. S6. Bind the ollama runner process to the optimal NUMA node combination and the CPU where the optimal NUMA node combination is located to avoid CPU inference phase accessing memory across NUMA nodes. S7, the ollama runner process calls llama.cpp to begin CPU-stage inference; S8 and llama.cpp begin the GPU-stage inference; S9 and llama.cpp return the inference results to the ollama runner process; S10, ollama runner returns the final inference result to the user terminal that executed ollama run.
2. The method according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41. Obtain the NUMAnode information of the node based on the / sys / devices / system / node / nodeX file of the node where ollama is located, where X is the code number 0, 1, 2, 3 and so on; S42. Obtain the CPU information on the NUMA node based on / sys / devices / system / node / nodeX / CPUX of the node where ollama is located, where X is the code number 0, 1, 2, 3 and so on; S43. Obtain the NUMA distance between nodes based on the / sys / devices / system / node / nodeX / distance file of the node where ollama is located, where X is the code number 0, 1, 2, 3 and so on.
3. The method according to claim 1, characterized in that, Step S6 specifically includes the following steps: Based on the optimal NUMA node combination obtained in step S5, bind the ollama runner process to the optimal NUMA node combination and the CPU where the optimal NUMA node combination is located.
4. The method according to claim 3, characterized in that, The ollama runner process is bound to the optimal NUMA node combination by calling the set_mempolicy system function.
5. The method according to claim 3, characterized in that, The ollama runner process is bound to the CPU containing the optimal NUMA node combination by calling the sched_setaffinity system function.