A large language model (LLM) device

Abstract

The application provides a virtual large language model LLM device, and belongs to the technical field of virtualization, and comprises an LLM app, a virtio_llm device, a virtio_llm driver, physical hardware Host, a virtual machine monitor Hypervisor, a virtual machine Guest and a large language model inference device LLM. The application does not rely on network transmission data, and does not need to passthrough computing resources to the virtual machine.

Description

Technical Field

[0001] This invention relates to Virtual Large Language Models (LLMs), and more particularly to a Virtual Large Language Model (LLM) device, belonging to the field of virtualization technology. Background Technology

[0002] With the rapid development of artificial intelligence technology, the potential of Large Language Models (LLMs) in various application fields is becoming increasingly significant. These applications, ranging from intelligent assistants to automatic content generation and complex question-answering systems, all place higher demands on computing resources. In particular, the training and inference of large models typically require powerful computing capabilities, such as graphics processing units (GPUs), and the demand for memory and network bandwidth is also increasing.

[0003] With the widespread adoption of virtualization technology, more and more applications are being deployed in virtual machines. However, many cloud service providers employ strict network isolation policies to ensure security and isolation. This restricts network access for virtual machines, making it difficult to directly call the APIs of artificial intelligence models within them. To overcome this limitation, developing a high-efficiency device capable of performing inference within virtual machines is particularly important.

[0004] Currently, in virtualization environments, virtual machines primarily rely on the following methods for performing inference:

[0005] 1. API Calls: Virtual machines can access AI models by calling external APIs, which typically involves network requests and responses. This method can utilize external GPU resources, but due to network isolation in cloud environments, it cannot be effectively implemented in most cases.

[0006] 2. Direct access to physical devices: Some virtualization platforms support passthrough from the physical GPU to the virtual machine, enabling the virtual machine to directly access hardware resources on the host machine. However, this method requires complex configuration and is not flexible enough in many cases, which may lead to uneven resource utilization.

[0007] 3. Predefined models: Some small machine learning models can be pre-installed and configured within the virtual machine. While this method can avoid the latency caused by external calls, it cannot meet the needs of complex tasks due to the limitation of model size.

[0008] The above method has the following specific disadvantages:

[0009] Network dependency: Most inference methods rely on stable network connections, and effective inference is almost impossible for virtual machines with network isolation.

[0010] Low resource utilization: While direct access to devices (such as GPUs) can improve performance, it usually requires complex configuration on the host machine, resulting in reduced resource sharing and utilization efficiency.

[0011] Insufficient flexibility: The use of predefined models limits reasoning capabilities and makes it impossible to dynamically adapt to different application needs.

[0012] Startup delay: When inference is implemented through API calls, the delay in network requests and responses may cause performance bottlenecks in application scenarios. Summary of the Invention

[0013] To address the above problems, this invention provides a Virtual Large Language Model (LLM) device.

[0014] To achieve the above objectives, the technical solution of the present invention is as follows: a virtual large language model (LLM) device, comprising an LLMapp, a virtio_llm device, a virtio_llm driver, and physical hardware (Host), a virtual machine monitor (Hypervisor), a virtual machine (Guest), and a large language model inference device (LLM), wherein,

[0015] The Hypervisor is deployed on physical hardware and is responsible for providing an abstraction layer between the physical hardware Host and the virtual machine Guest.

[0016] The virtual machine Guest is also deployed on physical hardware, running various operating systems and managed by the virtual machine monitor Hypervisor;

[0017] The physical hardware Host runs a Hypervisor in the virtualization environment and hosts virtual machines (Guest).

[0018] The Large Language Model Inference Device (LLM) runs on a physical hardware host and is responsible for performing inference tasks.

[0019] The LLM app is an application based on the Large Language Model Inference Device (LLM) and runs within the virtual machine Guest.

[0020] The virtio_llm device is a virtual LLM device created based on the Hypervisor. The virtio_llm device communicates with both the large language model inference device (LLM) and the virtio_llm driver.

[0021] The virtio_llm driver is a character device driver that runs within the virtual machine Guest and corresponds to the virtio_llmdevice device; the virtio_llm driver communicates with both the LLM app and the virtio_llm driver.

[0022] Furthermore, the physical hardware host is a physical machine or an operating system.

[0023] Furthermore, the LLM app is an intelligent assistant, a content generation system, or an automatic question-and-answer system.

[0024] Furthermore, the virtio_llm device is simulated as a PCI type device, and its configuration space includes the parameters required for inference.

[0025] Furthermore, the parameters required for the inference include: cue words, maximum generation length, temperature, top-level sampling, repetition penalty, and output format.

[0026] Furthermore, the prompt words are the initial text supplied to the Large Language Model Inference (LLM) device to guide the direction of the generated content;

[0027] The maximum generated length is the maximum length limit of the generated content;

[0028] The temperature is used to control the randomness of text generation, and its value ranges from 0 to 1.

[0029] The top-level sampling selects the next word from the k candidate words with the highest probability when generating each token;

[0030] The repetition penalty is used to penalize the model for generating the same word or phrase multiple times.

[0031] The output format types include question-and-answer, dialogue, or summary.

[0032] Furthermore, the virtio_llm driver is responsible for the following tasks: enumerating virtio_llm devices, initializing vring, encapsulating prompt words and related parameters into virtio protocol specification data frames, writing data frames, and reading data frames.

[0033] Furthermore, the write data frame refers to sending data to the virtio_llm device side; the read data frame refers to reading the inference result from the virtio_llm device side; and the vring is used to transfer data between the virtio_llm device and the virtio_llmdriver.

[0034] Furthermore, the inference process of the device includes the following:

[0035] 1) The virtio_llm driver enumerates the virtio_llm device;

[0036] 2) Write prompt words to the LLM app;

[0037] 3) The virtio_llm driver encapsulates the prompt words and inference parameters into a data frame according to the virtio protocol specification, and then sends the data frame to the virtio_llm device side through vring;

[0038] 4) After receiving the data frame, the virtio_llm device parses out the prompt words and inference parameters and inputs them into the host machine's Large Language Model Inference Device (LLM) for actual inference;

[0039] 5) After the virtio_llm device obtains the inference result, it also encapsulates the inference result into a data frame according to the virtio protocol specification and transmits it to the virtio_llm driver through vring;

[0040] 6) The LLM app finally obtains the inference results from / dev / virtio_llm through the read method.

[0041] Furthermore, step 1) includes the following steps:

[0042] a. Create the file / dev / virtio_llm for subsequent read and write operations;

[0043] b. Negotiation features between the virtio_llm driver and the virtio_llm device, including inference parameters of the device PCI configuration space;

[0044] c.vring initialization;

[0045] Step 2) The LLM app writes the prompt word via / dev / virtio_llm.

[0046] The beneficial effects of the virtual large language model (LLM) device of the present invention are as follows:

[0047] This invention does not rely on network data transmission but instead uses virtio for data transmission, thus solving the network dependency problem of existing technologies. This invention also eliminates the need to pass through computing resources to the virtual machine (Guest), resolving the issues of complex configuration and reduced resource utilization caused by existing methods that pass through computing resources to the virtual machine (Guest) and deploy large language model inference devices (LLMs) within the Virtual Machine (Guest).

[0048] This invention enables the virtual machine (Guest) to transmit prompt words and inference parameter configurations between itself and the host machine via the virtio protocol and a virtual queue. This method offers fast transmission speeds, eliminates the need for an open network, and enhances security.

[0049] The inference parameters of this invention are stored in the PCI configuration space of the virtual LLM device, making parameter changes more flexible. Attached Figure Description

[0050] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0051] Figure 1 This is an organizational chart of a Virtual Large Language Model (LLM) device according to the present invention;

[0052] Figure 2 This is a schematic diagram of the reasoning process implemented by the device of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0054] Definitions of abbreviations and key terms:

[0055] HYPERVISOR: A virtual machine monitor is a key component in virtualization technology. It is responsible for providing an abstraction layer between physical hardware (Host) and virtual machines (Guest), allowing multiple virtual machines to run in parallel on the same physical server.

[0056] GUEST: Guest refers to a virtual machine, which is managed by a virtual machine monitor (Hypervisor) and can run various operating systems.

[0057] HOST: In a virtualization environment, Host refers to the physical machine or operating system that runs the hypervisor and hosts virtual machines.

[0058] VIRTIO: Virtio is a virtualization standard used to provide efficient I / O between virtual machines and hosts. It defines a set of common devices and interfaces, facilitating the implementation of high-performance virtualization devices on different virtualization platforms.

[0059] VRING: Vring is a data structure in Virtio used to manage shared buffers between Virtio devices and virtual machines. It consists of a pair of descriptor rings and available / used rings.

[0060] The core idea of ​​this invention is to introduce a virtual large language model device (virtio_llm device) on the virtual machine Guest, and to introduce a driver for this device (virito_llm driver) within the virtual machine. Both transmit data via the virtio protocol specification and virtual queues. The Large Language Model Inference Device (LLM) application within the virtual machine Guest transmits inference prompts and related parameters to the virtio_llm device via the virtio_llm driver. The virtio_llm device then performs actual inference based on the prompts and parameters using the host machine's Large Language Model Inference Device (LLM). Finally, the inference result is returned to the virtual machine Guest, thus enabling inference to be completed within the virtual machine Guest. The following describes the process in conjunction with... Figure 1 and Figure 2 The present invention will be described in detail below.

[0061] Example 1

[0062] This embodiment combines Figure 1 This invention provides a detailed description of the structural composition of a Virtual Large Language Model (LLM) device, specifically:

[0063] The device of this invention includes an LLM app, a virtio_llm device, a virtio_llm driver, and physical hardware (Host), a hypervisor, a virtual machine (Guest), and a large language model inference device (LLM), wherein...

[0064] The Hypervisor is deployed on the physical hardware Host and is responsible for providing an abstraction layer between the physical hardware Host and the virtual machine Guest.

[0065] The virtual machine Guest is also deployed on the physical hardware Host, running various operating systems and managed by the virtual machine monitor Hypervisor;

[0066] The physical hardware host is specifically a physical machine or operating system that runs a hypervisor in a virtualization environment and hosts virtual machines (guests).

[0067] The Large Language Model Inference Device (LLM) runs on a physical hardware host and is responsible for performing inference tasks;

[0068] LLM apps are applications based on the Large Language Model Inference Device (LLM) that run within a virtual machine called Guest, and include intelligent assistants, content generation systems, and automatic question-answering systems.

[0069] The virtio_llm device is a virtual LLM device created based on the Hypervisor, simulating a PCI type device. Its configuration space includes the parameters required for inference. The virtio_llm device communicates with both the large language model inference device (LLM) and the virtio_llm driver.

[0070] The virtio_llm driver is a character device driver that runs within the virtual machine Guest and corresponds to the virtio_llmdevice device; the virtio_llm driver communicates with both the LLM app and the virtio_llm driver.

[0071] Further optimization,

[0072] The parameters required for inference included in the virtio_llm device configuration space are as follows:

[0073] 1. Prompt: The initial text provided to the large language model inference device to guide the direction of generated content;

[0074] 2. Maximum Length: The maximum length limit for the generated content;

[0075] 3. Temperature controls the randomness of text generation, and its value is usually between 0 and 1;

[0076] 4. Top-k Sampling: When generating each token, the next word is selected from the k candidate words with the highest probabilities;

[0077] 5. Repetition Penalty: This penalizes the model for generating the same word or phrase multiple times.

[0078] 6. Output Format: Specifies the type or style of the generated content, such as question and answer, dialogue, summary, etc.

[0079] The virtio_llm driver primarily performs the following tasks:

[0080] 1. Enumerate virtio_llm devices;

[0081] 2. Initialize vring;

[0082] 3. The encapsulation prompt and related parameters are in a virtio protocol specification data frame;

[0083] 4. Write data frames, i.e., send data to the virtio_llm device side;

[0084] 5. Read data frames, that is, read the inference results from the virtio_llm device side.

[0085] The aforementioned vring is used to transfer data between the virtio_llm device and the virtio_llm driver. The vrings on both sides of the virtio_llm device / driver are shared virtual queues that conform to the virtio protocol specification.

[0086] Example 2

[0087] Combination Figure 2 The process of performing inference from within a virtual machine using the device described in Embodiment 1 of the present invention is described in detail below:

[0088] 1. The virtio_llm driver enumeration includes virtio_llm devices, including:

[0089] a. Create the file / dev / virtio_llm for subsequent read and write operations;

[0090] b. Negotiation features between the virtio_llm driver and the virtio_llm device, including inference parameters for the device PCI configuration space;

[0091] c.vring initialization.

[0092] 2. The LLM app writes the prompt word via / dev / virtio_llm.

[0093] 3. The virtio_llm driver encapsulates the prompt words and inference parameters into a data frame according to the virtio protocol specification, and then sends the data frame to the virtio_llm device side via vring.

[0094] 4. After receiving the data frame, the virtio_llm device parses out the prompt words and inference parameters and inputs them into the host machine's Large Language Model Inference (LLM) device for actual inference.

[0095] 5. After obtaining the inference results, the virtio_llm device also encapsulates the inference results into data frames according to the virtio protocol specification and transmits them to the virtio_llm driver via vring.

[0096] 6. The LLM app finally obtains the inference results from / dev / virtio_llm using the read method.

[0097] In summary, this invention enables virtual machines to access the host machine's Large Language Model Inference (LLM) device to perform inference without an open network; this invention does not require the virtual machine to have an open network, thus avoiding network attacks and making the virtual machine more secure; this invention does not require passing computing resources to the virtual machine, thus avoiding problems such as complex configuration and reduced resource utilization.

[0098] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A virtual large language model (LLM) device, virtio_llm device, characterized in that, The components associated with the virtual large language model (LLM) device (virtio_llm device) include the LLM app, the virtio_llm driver, the physical hardware host, the hypervisor, the virtual machine guest, and the large language model inference device (LLM). The virtual machine guest is network isolated from the physical hardware host. The Hypervisor is deployed on physical hardware and is responsible for providing an abstraction layer between the physical hardware Host and the virtual machine Guest, as well as managing and emulating the virtual machine Guest. The virtual machine Guest is deployed on top of the Hypervisor, running various operating systems and managed by the Hypervisor. The Large Language Model Inference Device (LLM) runs on a physical hardware host and is responsible for performing the actual inference tasks. The LLM app is an application developed based on the virtual large language model LLM device virtio_llm device. It runs in the virtual machine Guest and serves as an intelligent assistant, content generation system, or automatic question answering system. The virtual large language model LLM device, virtio_llm device, runs in the Hypervisor layer; The virtio_llm driver runs within the virtual machine Guest and is the driver corresponding to the virtual large language model LLM device virtio_llm device. The LLM app, the virtual large language model LLM device (virtio_llm device), and the virtio_llm driver together construct the inference path.

2. The virtual large language model (LLM) device virtio_llm device according to claim 1, characterized in that, The physical hardware host is a physical machine or an operating system.

3. The virtual large language model (LLM) device virtio_llm device according to claim 1, characterized in that, The virtual large language model (LLM) device (virtio_llm device) is used to solve the problem that LLM apps within the virtual machine (Guest) cannot access the large model in scenarios where there is network isolation between the virtual machine (Guest) and the physical hardware (Host).

4. The virtual large language model (LLM) device virtio_llm device according to claim 1, characterized in that, The virtual large language model LLM device, virtio_llm device, is a PCI type device whose configuration space includes the parameters required for inference.

5. The virtual large language model (LLM) device virtio_llm device according to claim 4, characterized in that, The parameters required for the inference include: cue words, maximum generation length, temperature, top-level sampling, repetition penalty, and output format.

6. The virtual large language model (LLM) device virtio_llm device according to claim 1, characterized in that, The virtio_llm driver is responsible for the following tasks: enumerating virtio_llm devices, encapsulating prompts and related inference parameters from the LLM app into virtio protocol specification data frames and forwarding them to the virtual large language model LLM device virtio_llm device, and parsing virtio protocol specification data frames from the virtual large language model LLM device virtio_llm device and forwarding them to the LLM app.

7. The virtual large language model (LLM) device virtio_llm device according to claim 1, characterized in that, The virtual large language model LLM device parses the virtio protocol specification data frame containing prompt words and inference parameters from the virtio_llm driver and forwards it to the large language model inference device LLM in the physical hardware host. It also encapsulates the inference result from the large language model inference device LLM in the physical hardware host into a virtio protocol specification data frame and forwards it to the virtio_llm driver.

Images