A program running method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a program running method and device, electronic equipment and storage medium, and relate to the technical field of computer, which can save communication overhead and optimize resource configuration of different processes. The method comprises the following steps: obtaining at least two virtual accelerator cards on a same physical accelerator card, the at least two virtual accelerator cards comprising a first virtual accelerator card and a second virtual accelerator card, the first virtual accelerator card comprising a first virtual computing unit and a first virtual video memory, the second virtual accelerator card comprising a second virtual computing unit and a second virtual video memory, the first virtual computing unit and the second virtual computing unit being isolated from each other, and the first virtual video memory and the second virtual video memory being shared with each other and mapped to a real physical video memory space; running a first process based on the first virtual accelerator card and running a second process based on the second virtual accelerator card, and the first process and the second process share data through the first virtual video memory and the second virtual video memory during running. The present application is suitable for optimization of program running.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a program execution method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of computer and artificial intelligence technologies, the types and complexity of services have increased dramatically. A service often requires multiple processes to work together to complete it. These processes may be in a parallel relationship, a serial relationship, or a combination of both.

[0003] Since different processes often require significantly different resources during operation, in order to make the resource allocation of different processes more reasonable, different processes are often distributed to run on different accelerator cards. However, this introduces a large inter-card communication overhead. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a program running method, apparatus, electronic device, and storage medium that can optimize resource allocation for different processes while saving communication overhead.

[0005] In a first aspect, embodiments of the present invention provide a program execution method, comprising: acquiring at least two virtual accelerator cards on the same physical accelerator card, the at least two virtual accelerator cards including a first virtual accelerator card and a second virtual accelerator card, wherein the first virtual accelerator card includes a first virtual computing unit and a first virtual video memory, the second virtual accelerator card includes a second virtual computing unit and a second virtual video memory, the first virtual computing unit and the second virtual computing unit are isolated from each other, and the first virtual video memory and the second virtual video memory share the real physical video memory space mapped to each other; running a first process based on the first virtual accelerator card, and running a second process based on the second virtual accelerator card, wherein the first process and the second process share data through the first virtual video memory and the second virtual video memory during execution.

[0006] In one implementation, acquiring at least two virtual accelerator cards on the same physical accelerator card includes: virtualizing the physical computing units of the target physical accelerator card into at least two virtual computing units, the at least two virtual computing units including a first virtual computing unit and a second virtual computing unit, the first virtual computing unit and the second virtual computing unit being established based on different hardware resources on the target physical accelerator card and isolated from each other; virtualizing the real physical video memory space of the target physical accelerator card into at least two virtual video memories, the at least two virtual video memories including the first virtual video memory and the second virtual video memory; the first virtual video memory corresponding to the first virtual computing unit, and the second virtual video memory corresponding to the second virtual computing unit; the first virtual video memory and the second virtual video memory uniformly performing virtual physical video memory space to real physical video memory space mapping; the virtual physical video memory space of the first virtual video memory being mapped to a first region of the real physical video memory space, the virtual physical video memory space of the second virtual video memory being mapped to a second region of the real physical video memory space, the first region being the same as the second region; the first process and the second process sharing data through the first region of the real physical video memory space during operation.

[0007] In one implementation, the first region and the second region are default regions of the real physical video memory space, or the first region and the second region are configured by a virtual mask register in a preset virtual device model.

[0008] In one embodiment, after virtualizing the real physical video memory space of the target physics accelerator card into at least two virtual video memories, the method further includes: in the driver program of the target physics accelerator card, setting the first virtual accelerator card to have management permissions for a third region of the real physical video memory space, and setting the second virtual accelerator card to have management permissions for a fourth region of the real physical video memory space, wherein the third region is different from the fourth region; both the third region and the fourth region are located within the first region; the management permissions include allocating video memory space and / or reclaiming video memory space.

[0009] In one implementation, the first virtual video memory and the second virtual video memory are uniformly mapped from virtual physical video memory space to real physical video memory space. The first virtual video memory is visible to a first region of the real physical video memory space, and the second virtual video memory is visible to a second region of the real physical video memory space. The first region and the second region are the same. The first process and the second process share data through the first region of the real physical video memory space during operation.

[0010] In one implementation, running a first process based on the first virtual accelerator card and running a second process based on the second virtual accelerator card includes: running the first process based on the first virtual accelerator card and writing the first data generated during the running of the first process into the first region of the real physical video memory space through the first virtual video memory; running the second process based on the second virtual accelerator card and reading the first data from the first region of the real physical video memory space through the second virtual video memory for processing by the second process.

[0011] In one implementation, the first virtual accelerator card has management authority over a third region of the real physical video memory space, and the second virtual accelerator card has management authority over a fourth region of the real physical video memory space, wherein the third region is different from the fourth region; both the third region and the fourth region are located within the first region; the management authority includes allocating video memory space and / or reclaiming video memory space; the step of running the first process based on the first virtual accelerator card and writing the first data generated during the running of the first process into the first region of the real physical video memory space through the first virtual video memory includes: running the first process based on the first virtual accelerator card, obtaining a data write request from the first process to write the first data into the real physical video memory space; querying the address range in the driver of the physical accelerator card that the first virtual accelerator card has management authority according to the data write request to obtain the third region; allocating a corresponding data storage area for the data write request in the third region; and writing the first data into the data storage area.

[0012] In one implementation, running the second process based on the second virtual accelerator card and reading the first data from the first region of the real physical video memory space via the second virtual video memory for processing by the second process includes: running the second process based on the second virtual accelerator card; obtaining a data read request from the second process, the data read request carrying the target video memory address of the data storage region to be read; setting the data storage region to a shareable state according to the data read request, so that the second virtual accelerator card can read the first data from the target video memory address; incrementing the reference count of the target video memory address by 1; and decrementing the reference count of the target video memory address by 1 after the first data is read.

[0013] In one embodiment, after allocating a corresponding data storage area for the data write request in the third region, the method further includes: obtaining a request to reclaim the data storage area corresponding to the target memory address; reclaiming the data storage area when the reference count of the target memory address is 0; or, temporarily suspending the reclamation of the data storage area when the reference count of the target memory address is greater than 0.

[0014] In one embodiment, the number of the first virtual accelerator cards is at least one, and the number of the second virtual accelerator cards is at least one.

[0015] In one implementation, the first process includes a process executed by the large language model during the pre-filling phase, and the second process includes a process executed by the large language model during the decoding phase.

[0016] Secondly, embodiments of the present invention also provide a program running apparatus, comprising: an acquisition module, configured to acquire at least two virtual accelerator cards on the same physical accelerator card, the at least two virtual accelerator cards including a first virtual accelerator card and a second virtual accelerator card, wherein the first virtual accelerator card includes a first virtual computing unit and a first virtual video memory, the second virtual accelerator card includes a second virtual computing unit and a second virtual video memory, the first virtual computing unit and the second virtual computing unit are isolated from each other, and the first virtual video memory and the second virtual video memory share the real physical video memory space mapped to each other; and a running module, configured to run a first process based on the first virtual accelerator card and run a second process based on the second virtual accelerator card, wherein the first process and the second process share data through the first virtual video memory and the second virtual video memory during operation.

[0017] In one embodiment, the acquisition module includes: a first virtualization submodule, configured to virtualize the physical computing units of the target physics acceleration card into at least two virtual computing units, the at least two virtual computing units including a first virtual computing unit and a second virtual computing unit, the first virtual computing unit and the second virtual computing unit being established based on different hardware resources on the target physics acceleration card and isolated from each other; a second virtualization submodule, configured to virtualize the real physical video memory space of the target physics acceleration card into at least two virtual video memories, the at least two virtual video memories including the first virtual video memory and the second virtual video memory; the first virtual video memory corresponds to the first virtual computing unit, and the second virtual video memory corresponds to the second virtual computing unit; the first virtual video memory and the second virtual video memory are uniformly mapped from virtual physical video memory space to real physical video memory space; the virtual physical video memory space of the first virtual video memory is mapped to a first region of the real physical video memory space, and the virtual physical video memory space of the second virtual video memory is mapped to a second region of the real physical video memory space, the first region and the second region being the same; the first process and the second process share data through the first region of the real physical video memory space during operation.

[0018] In one implementation, the first region and the second region are default regions of the real physical video memory space, or the first region and the second region are configured by a virtual mask register in a preset virtual device model.

[0019] In one embodiment, the acquisition module further includes: a setting submodule, configured to, after virtualizing the real physical video memory space of the target physics accelerator card into at least two virtual video memories, in the driver program of the target physics accelerator card, set the first virtual accelerator card to have management permissions for a third region of the real physical video memory space, and set the second virtual accelerator card to have management permissions for a fourth region of the real physical video memory space, wherein the third region is different from the fourth region; both the third region and the fourth region are located within the first region; the management permissions include allocating video memory space and / or reclaiming video memory space.

[0020] In one implementation, the first virtual video memory and the second virtual video memory are uniformly mapped from virtual physical video memory space to real physical video memory space. The first virtual video memory is visible to a first region of the real physical video memory space, and the second virtual video memory is visible to a second region of the real physical video memory space. The first region and the second region are the same. The first process and the second process share data through the first region of the real physical video memory space during operation.

[0021] In one embodiment, the running module includes: a first running submodule, configured to run the first process based on the first virtual accelerator card, and write the first data generated during the running of the first process into the first region of the real physical video memory space through the first virtual video memory; and a second running submodule, configured to run the second process based on the second virtual accelerator card, and read the first data from the first region of the real physical video memory space through the second virtual video memory for processing by the second process.

[0022] In one implementation, the first virtual accelerator card has management authority over a third region of the real physical video memory space, and the second virtual accelerator card has management authority over a fourth region of the real physical video memory space, wherein the third region is different from the fourth region; both the third region and the fourth region are located within the first region; the management authority includes allocating video memory space and / or reclaiming video memory space; the first running submodule is specifically configured to: run the first process based on the first virtual accelerator card, obtain a data write request from the first process to write the first data to the real physical video memory space; according to the data write request, query the address range in the driver of the physical accelerator card where the first virtual accelerator card has management authority to obtain the third region; allocate a corresponding data storage area for the data write request in the third region; and write the first data to the data storage area.

[0023] In one implementation, the second running submodule is specifically configured to: run the second process based on the second virtual accelerator card; obtain a data read request from the second process, wherein the data read request carries the target video memory address of the data storage area to be read; set the data storage area to a shareable state according to the data read request, so that the second virtual accelerator card reads the first data from the target video memory address; increment the reference count of the target video memory address by 1; and decrement the reference count of the target video memory address by 1 after the first data is read.

[0024] In one implementation, the first running submodule is further configured to: after allocating a corresponding data storage area for the data write request in the third region, obtain a request to reclaim the data storage area corresponding to the target video memory address; reclaim the data storage area when the reference count of the target video memory address is 0; or, postpone the reclamation of the data storage area when the reference count of the target video memory address is greater than 0.

[0025] In one embodiment, the number of the first virtual accelerator cards is at least one, and the number of the second virtual accelerator cards is at least one.

[0026] In one implementation, the first process includes a process executed by the large language model during the pre-filling phase, and the second process includes a process executed by the large language model during the decoding phase.

[0027] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: a processor and a memory, the processor being electrically connected to the memory; the memory being used to store executable program code; the processor running a program corresponding to the executable program code by reading the executable program code stored in the memory, thereby implementing any program running method provided by embodiments of the present invention.

[0028] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement any of the program execution methods provided in the embodiments of the present invention.

[0029] The program execution method, apparatus, electronic device, and computer-readable storage medium provided by embodiments of the present invention can obtain at least two virtual accelerator cards on the same physical accelerator card, run a first process based on a first virtual accelerator card, and run a second process based on a second virtual accelerator card. Since the first virtual computing unit of the first virtual accelerator card and the second virtual computing unit of the second virtual accelerator card are isolated from each other, the required computing resources can be allocated to the first and second processes according to their characteristics, making the allocation of computing resources between different processes more reasonable. Furthermore, since the first and second virtual accelerator cards are located on the same physical accelerator card, the first virtual video memory of the first virtual accelerator card and the second virtual video memory of the second virtual accelerator card share the mapped real physical video memory space. Thus, the first and second processes can obtain the required storage resources and share data based on the shared real physical video memory space without needing to transmit data through time-consuming inter-card communication. Therefore, resource allocation between different processes can be optimized while saving communication overhead. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart of a program execution method provided for an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the working principle of a large language model in an embodiment of the present invention; Figure 3 This is a schematic diagram of PD separation based on a virtual accelerator card in an embodiment of the present invention; Figure 4 This is a schematic diagram of a structure of the virtual device model, accelerator card, accelerator card driver, and application framework layer in an embodiment of the present invention; Figure 5 A detailed flowchart of a program execution method provided for an embodiment of the present invention; Figure 6 A schematic diagram of a program execution device provided for an embodiment of the present invention; Figure 7 A schematic diagram of an electronic device provided as an embodiment of the present invention. Detailed Implementation

[0032] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0034] In a first aspect, embodiments of the present invention provide a program execution method that can optimize resource allocation for different processes while saving communication overhead.

[0035] like Figure 1 As shown, an embodiment of the present invention provides a program execution method that may include: S11, acquire at least two virtual accelerator cards on the same physical accelerator card, the at least two virtual accelerator cards include a first virtual accelerator card and a second virtual accelerator card, wherein the first virtual accelerator card includes a first virtual computing unit and a first virtual video memory, the second virtual accelerator card includes a second virtual computing unit and a second virtual video memory, the first virtual computing unit and the second virtual computing unit are isolated from each other, and the first virtual video memory and the second virtual video memory share the real physical video memory space mapped to each other; An accelerator card is a standalone hardware card that plugs into a server's PCIe slot. It integrates a dedicated parallel computing chip, dedicated onboard video memory, and high-speed interconnect interfaces. It is used to offload massive parallel computing tasks from the CPU, improving throughput and reducing CPU load in specific scenarios. This accelerator card, referred to as a physical accelerator card in this article, is used to distinguish it from a virtual accelerator card.

[0036] In embodiments of the present invention, based on virtualization technology, one physical accelerator card can be virtualized into at least two virtual accelerator cards. These virtual accelerator cards may include, for example, a first virtual accelerator card and a second virtual accelerator card. Virtualization technology is a technology that abstracts a single hardware resource, enabling it to be exposed to software in multiple forms. For example, through virtualization technology, one physical machine can be virtualized into multiple machines, running multiple operating systems. Similarly, one physical device can be virtualized into multiple devices for use.

[0037] Similar to a physical accelerator card, each virtual accelerator card can include virtual computing units and virtual video memory. The virtual computing units on the first virtual accelerator card are called first virtual computing units, and the virtual computing units on the second virtual accelerator card are called second virtual computing units. The virtual video memory on the first virtual accelerator card is called first virtual video memory, and the virtual video memory on the second virtual accelerator card is called second virtual video memory.

[0038] In embodiments of the present invention, the first virtual computing unit and the second virtual computing unit are isolated from each other, meaning that the first virtual computing unit and the second virtual computing unit can each execute computing tasks independently. The first virtual video memory and the second virtual video memory share the real physical video memory space mapped to each other; that is, the real physical video memory space mapped to the first virtual video memory is the same as the real physical video memory space mapped to the second virtual video memory.

[0039] S12, a first process is run based on the first virtual accelerator card, and a second process is run based on the second virtual accelerator card, wherein the first process and the second process share data through the first virtual video memory and the second virtual video memory during operation.

[0040] After obtaining the first and second virtual accelerator cards, in this step, a first process can run on the first virtual accelerator card, and a second process can run on the second virtual accelerator card. The first and second processes can share data, cooperate, and jointly complete service tasks during operation. Since the real physical memory space mapped to the first virtual memory is the same as that mapped to the second virtual memory, data generated or used by the first process can be read by the second process through the shared real physical memory space, and similarly, data generated or used by the second process can also be read by the first process through the shared real physical memory space. This eliminates the need for time-consuming inter-card communication to transfer data, thus effectively saving communication overhead.

[0041] The program execution method provided by the embodiments of the present invention can acquire at least two virtual accelerator cards on the same physical accelerator card, run a first process based on the first virtual accelerator card, and run a second process based on the second virtual accelerator card. Since the first virtual computing unit of the first virtual accelerator card and the second virtual computing unit of the second virtual accelerator card are isolated from each other, the computing resources required by the first and second processes can be allocated according to their characteristics, making the allocation of computing resources between different processes more reasonable. Furthermore, since the first and second virtual accelerator cards are located on the same physical accelerator card, the first virtual video memory of the first virtual accelerator card and the second virtual video memory of the second virtual accelerator card share the mapped real physical video memory space. Thus, the first and second processes can obtain the required storage resources and share data based on the shared real physical video memory space without needing to transmit data through time-consuming inter-card communication. Therefore, it can optimize the resource allocation of different processes while saving communication overhead.

[0042] Specifically, in the embodiments of the present invention, the number of first virtual accelerator cards and second virtual accelerator cards in the same physical accelerator card can be at least one. Each first virtual accelerator card can run one first thread, and each second virtual accelerator card can run one second thread. The specific number of first virtual accelerator cards and second virtual accelerator cards can be set or adjusted according to requirements.

[0043] Because different services have different program structures, the resource consumption of the first thread and the second thread differs. Therefore, the optimal resource configuration may result in different ratios between the first thread and the second thread. For example, in one example, the number of first virtual accelerator cards and the number of second virtual accelerator cards can both be one; in another example, the number of first virtual accelerator cards and the number of second virtual accelerator cards can both be multiple. Of course, the number of first virtual accelerator cards can be one, and the number of second virtual accelerator cards can be multiple, or the number of first virtual accelerator cards can be multiple, and the number of second virtual accelerator cards can be one. The embodiments of this invention do not limit this.

[0044] Furthermore, in addition to the first and second virtual accelerator cards, the same physical accelerator card can also include other virtual accelerator cards, and other processes can run on these other virtual accelerator cards. For example, it can also include a third virtual accelerator card, on which a third process can run. The third virtual accelerator card can have similar characteristics to the first or second virtual accelerator card, which will not be elaborated here.

[0045] In embodiments of the present invention, the first process and the second process can be various processes that interact and collaborate with each other, and the embodiments of the present invention do not limit this. For example, in one example, the first process may include the process executed by the large language model in the prefill (P) stage, and the second process may include the process executed by the large language model in the decoding (D) stage.

[0046] With the increasing prevalence of AI applications based on Large Language Models (LLMs), improving the performance and efficiency of LLM inference is becoming extremely important. Currently, most mainstream LLMs employ a transformer architecture: for user input, they compute output based on an attention mechanism. Its simplified architecture is as follows: Figure 2 As shown. Specifically, the large-scale model inference process based on the transformer architecture can be divided into two stages: prefill and decode. In the prefill stage, the first output term is calculated based on all user input prompts, and the key-value vector of each prompt is saved in a cache called the KV cache. This vector value is reused during the decode process. The decode process is similar to prefill, except that it iterates: each iteration produces one output term, and the new output term and the previous overall output sequence serve as input for the next iteration, until all terms are generated.

[0047] This characteristic of the Transformer model leads to significant differences in the operational features of the prefill and decode phases. The prefill phase processes all user input prompts at once, involving numerous matrix operations and demanding high computational power, especially for long text inputs. In contrast, after the prefill phase, the decode phase produces only one output per iteration, and all iterations must be completed sequentially, thus reducing computational power requirements. However, as the number of outputs increases, the memory pressure on the GPU used to store the KVcache continuously increases. A comparison of the characteristics of the prefill and decode phases is shown in Table 1.

[0048] Table 1 Runtime short long computational load high Low Video memory usage Relatively low Relatively high This leads to insufficient resource utilization when running Transformer-based LLM inference on a single AI accelerator card. The decoding phase, which generates output sequentially with low parallelism, is prone to idle computing power, thus impacting overall throughput.

[0049] Therefore, in one embodiment of the present invention, prefill and decode are decoupled into a first process and a second process, and the first process and the second process are run on different virtual accelerator cards formed by the same physical accelerator card. Different computing resources are configured according to the different characteristics of the two stages, and they share video memory resources, thereby improving the overall resource utilization and effectively enhancing model performance. For example, the prefill stage can be processed in parallel by M first processes, and the decoding stage can be processed in parallel by N first processes, where M and N are both positive integers. In one implementation, M is less than N.

[0050] In one implementation, under the PD separation scheme, the deployment of the first thread and the second thread in the virtual accelerator card can be as follows: Figure 3 As shown in the diagram. The first thread runs on the first virtual accelerator card, and the second thread runs on the second virtual accelerator card. For example, in one implementation, because the parallelism of single-path processing in the D stage is low and the operation is slow, multiple second processes running in the D stage can be configured for one first process running in the P stage to process multiple requests in parallel, thereby matching the processing speed of the D stage with that of the P stage. For example, in one example, the ratio of the first process to the second process can be 1:3, and correspondingly, the ratio of the number of first virtual accelerator cards to the number of second accelerator cards can also be 1:3.

[0051] Of course, the DP separation of the large language model is only one specific application scenario of the program running method provided by the embodiments of the present invention. The program running method provided by the embodiments of the present invention can also be used in other scenarios, and the embodiments of the present invention do not limit this.

[0052] The following provides a detailed explanation of how to obtain the virtual first accelerator card and the second accelerator card, and how to run the corresponding processes on the first accelerator card and the second accelerator card in the program running method provided by the embodiments of the present invention.

[0053] Specifically, in one embodiment of the present invention, step S11, obtaining at least two virtual accelerator cards on the same physical accelerator card, may include: virtualizing the physical computing unit of the target physical accelerator card into at least two virtual computing units, the at least two virtual computing units including a first virtual computing unit and a second virtual computing unit, the first virtual computing unit and the second virtual computing unit being established based on different hardware resources on the target physical accelerator card and isolated from each other; virtualizing the real physical video memory space of the target physical accelerator card into at least two virtual video memories, the at least two virtual video memories including the first virtual video memory and the second virtual video memory; the first virtual video memory corresponding to the first virtual computing unit, and the second virtual video memory corresponding to the second virtual computing unit; the first virtual video memory and the second virtual video memory uniformly performing a mapping from virtual physical video memory space to real physical video memory space; the virtual physical video memory space of the first virtual video memory being mapped to a first region of the real physical video memory space, the virtual physical video memory space of the second virtual video memory being mapped to a second region of the real physical video memory space, the first region being the same as the second region; the first process and the second process sharing data through the first region of the real physical video memory space during operation.

[0054] In embodiments of the present invention, a physical accelerator card may include a large number of physical computing units. Each physical computing unit may include vector pipelines, scalar pipelines, local shared memory, registers, etc., supporting parallel computing. In this embodiment, virtualization technology can be used to virtualize the physical computing units on the same physical accelerator card into at least two virtual computing units. Different virtual computing units can be mapped to different physical computing units, thereby achieving mutual isolation. For example, in one implementation, the at least two virtual computing units may include a first virtual computing unit and a second virtual computing unit. The first virtual computing unit and the second virtual computing unit can be established and isolated from each other based on different hardware resources on the target physical accelerator card, for example, they can be established and isolated from each other based on different physical computing units on the target physical accelerator card.

[0055] In addition to virtualizing the physical computing units in the target physics accelerator card into virtual computing units, the real physical memory space in the target physics accelerator card can also be virtualized into multiple virtual physical memory spaces. Here, real physical memory space can refer to the actual physical memory space of the target physics accelerator card, while virtual physical memory space can refer to the physical memory space from the perspective of each virtual accelerator card, that is, the physical memory space that each virtual accelerator card believes it possesses. However, in essence, virtual physical memory space is still virtual. Virtual physical memory space and real physical memory space can have a corresponding relationship; the virtual physical memory space of each virtual accelerator card can be mapped to a portion or all of the real physical memory space of the target physics accelerator card.

[0056] For example, in one implementation, the at least two virtual video memories obtained through physical video memory virtualization may include a first virtual video memory and a second virtual video memory; wherein, the first virtual video memory corresponds to a first virtual computing unit, and the two can form a first virtual accelerator card, and the second virtual video memory corresponds to a second virtual computing unit, and the two can form a second virtual accelerator card. The first virtual video memory and the second virtual video memory can uniformly map virtual physical video memory space to real physical video memory space, that is, the first virtual video memory and the second virtual video memory use the same virtual physical video memory space and are mapped to the same real physical video memory space in the same way. In this way, the same virtual physical address in different virtual video memories will be mapped to the same real physical address. For example, virtual physical address A1 in the first virtual video memory and virtual physical address A1 in the second virtual video memory will both be mapped to real physical address B1.

[0057] Specifically, in one embodiment of the present invention, the virtual physical memory space of the first virtual video memory can be mapped to a first region of the real physical video memory space, and the virtual physical memory space of the second virtual video memory can be mapped to a second region of the real physical video memory space, with the first region and the second region being the same; that is, the virtual physical memory space of the first virtual video memory and the virtual physical memory space of the second virtual video memory are both mapped to the same region of the real physical video memory space, namely the first region (which is also the second region). In this way, the first virtual accelerator card and the second virtual accelerator card can use the same region of the real physical video memory space, thereby realizing video memory sharing. Based on this, when the first process and the second process are running in step S12, the first process and the second process can share data through the first region of the real physical video memory space during operation.

[0058] In embodiments of the present invention, the first region and the second region can be the same, and the specific sizes of the first region and the second region are not limited. For example, in one example, the first region and the second region can be a part of the actual physical video memory space, and in another example, the first region and the second region can be the entire area of ​​the actual physical video memory space.

[0059] Furthermore, the first region and the second region can be either fixed or configurable, and the embodiments of the present invention do not limit this. For example, in one example, the first region and the second region can be the default regions of the real physical video memory space; in another example, the first region and the second region can be configured by a virtual mask register in a preset virtual device model.

[0060] Here, the virtual device model is a set of rules or architectural models used to define how to abstract a physical device into multiple logically independent virtual devices. It serves as a template for hardware virtualization, and the virtual device model must be followed when virtualizing accelerator cards to specifically implement virtual device instances. For example, in one implementation, the relationship between the virtual device model and the AI ​​(Artificial Intelligence) accelerator card, the AI ​​accelerator card driver, and the AI ​​framework layer software can be as follows: Figure 4 As shown.

[0061] In one implementation, the virtual device model may include a virtual mask register. Each virtual mask register can correspond one-to-one with a virtual memory location. Each virtual mask register may contain multiple bits, and each bit can be set to 0 or 1 to control whether the virtual memory is visible to a specific address block in the real physical memory space. For example, in one scenario, the real physical memory space is divided into multiple address blocks in 1024KB units. Each address block corresponds to a bit in the virtual mask register. If the bit is 0, it means the virtual memory is not visible to that address block; if the bit is 1, it means the virtual memory is visible to that address block. Thus, by setting the virtual mask registers of different virtual memory locations to the same value, each virtual memory location can be mapped to the same region of the real physical memory space.

[0062] In embodiments of the present invention, the first virtual video memory and the second virtual video memory can uniformly map virtual physical video memory space to real physical video memory space. The first virtual video memory can be visible to a first region of the real physical video memory space, and the second virtual video memory can be visible to a second region of the real physical video memory space. The first region and the second region are the same. Based on this, the first process and the second process can share data through the first region of the real physical video memory space during operation.

[0063] Specifically, in one implementation, running a first process based on the first virtual accelerator card and running a second process based on the second virtual accelerator card may include: running the first process based on the first virtual accelerator card and writing first data generated during the execution of the first process into a first region of the real physical video memory space via the first virtual video memory; and running the second process based on the second virtual accelerator card and reading first data from the first region of the real physical video memory space via the second virtual video memory for processing by the second process. That is, the first data generated by the first process running on the first virtual accelerator card can be written into the first region of the real physical video memory space via the first virtual video memory, and the second process, when running on the second virtual accelerator card, can read the first data from the first region via the second virtual video memory and process it.

[0064] In the foregoing embodiments, after mapping the virtual physical memory space of the virtual video memory to the first region of the real physical memory space of the physical video memory, the virtual video memory becomes visible to the first region of the real physical memory space. The virtual accelerator card can then perform read and / or write operations on the first region of the real physical memory space through the virtual physical video memory space. However, the embodiments of the present invention are not limited to this. In other embodiments of the present invention, a dedicated management region can be further divided for each virtual accelerator card within the first region to better manage video memory resources and prevent memory resource encroachment between different virtual accelerator cards.

[0065] Specifically, in one embodiment of the present invention, after virtualizing the real physical video memory space of the target physics accelerator card into at least two virtual video memories, the program running method provided by the embodiment of the present invention may further include: in the driver program of the target physics accelerator card, setting the first virtual accelerator card to have management authority over a third region of the real physical video memory space, and setting the second virtual accelerator card to have management authority over a fourth region of the real physical video memory space, wherein the third region is different from the fourth region; both the third region and the fourth region are located within the first region; the management authority includes allocating video memory space and / or reclaiming video memory space.

[0066] For example, in one instance, both the first and second virtual accelerator cards are mapped to a first region (address 0-4G) of the real physical video memory space. In the driver for the target physical accelerator card, the first virtual accelerator card is further configured to have management permissions for the portion of the first region with addresses 0-2G; this portion with addresses 0-2G then becomes the third region. Similarly, the first virtual accelerator card can be configured to have management permissions for the portion of the first region with addresses 2G-4G; this portion with addresses 2G-4G then becomes the fourth region.

[0067] Based on this, in the aforementioned embodiments, running the first process based on the first virtual accelerator card and writing the first data generated during the execution of the first process into the first region of the real physical video memory space via the first virtual video memory can specifically include: running the first process based on the first virtual accelerator card, obtaining a data write request from the first process to write the first data into the real physical video memory space; according to the data write request, querying the address range with management authority of the first virtual accelerator card in the driver of the physical accelerator card to obtain the third region; allocating a corresponding data storage region for the data write request in the third region; and writing the first data into the data storage region. Thus, when the first process needs to write the first data to the video memory, it can first query the driver to see which region of the real physical video memory space the first virtual accelerator card has management authority over, and allocate video memory for the first data from the region with management authority, thereby managing the video memory resources.

[0068] Furthermore, in one embodiment of the present invention, after the first process writes the first data into a first region of the real physical video memory space, the second process running on the second virtual accelerator card can read the first data from the first region. Specifically, in one implementation, running the second process based on the second virtual accelerator card and reading the first data from the first region of the real physical video memory space through the second virtual video memory for processing by the second process may include: running the second process based on the second virtual accelerator card; obtaining a data read request from the second process, the data read request carrying the target video memory address of the data storage region to be read; setting the data storage region to a shareable state according to the data read request, so that the second virtual accelerator card can read the first data from the target video memory address; incrementing the reference count of the target video memory address by 1; and decrementing the reference count of the target video memory address by 1 after the first data is read.

[0069] It should be noted that in the embodiments of the present invention, the data storage area is allocated from the third area, which is a part of the first area. Furthermore, the first area itself is the real physical video memory space shared by the first and second virtual accelerator cards. Therefore, even if the data storage area is not set to a shareable state, it is still shared by the first and second virtual accelerator cards. The reason for setting the data storage area to a shareable state is to allow for more granular control through software, thereby further improving the reliability of program operation. For example, if an allocated video memory area is declared as shareable, then when reclaiming that area, it is necessary to determine whether it can be reclaimed based on whether its reference count is 0. However, if an allocated video memory area is not declared as shareable, then when reclaiming that area, it is not necessary to consider the usage of that area by other processes; it can be reclaimed directly.

[0070] Specifically, in this embodiment, a reference count variable can be maintained for the target video memory address. When a second process running on the second virtual accelerator card reads the first data, it can be considered a reference to the target video memory address. Different second processes reading the first data can be considered different references to the target video memory address. Each time a second process reads the first data, the reference count is incremented by 1. After a second process finishes reading the first data, the reference count is decremented by 1. In this way, the reference count can be used to determine whether other processes are currently reading the first data. For example, if the reference count is 0, it means that no other process is currently reading the first data; if the reference count is greater than 0, it means that other processes are currently reading the first data.

[0071] Based on this, in one embodiment, after allocating a corresponding data storage area for the data write request in the third area, the program running method provided by the embodiments of the present invention may further include: obtaining a request to reclaim the data storage area corresponding to the target video memory address; reclaiming the data storage area when the reference count of the target video memory address is 0; or, postponing the reclamation of the data storage area when the reference count of the target video memory address is greater than 0. In this way, when the first virtual accelerator card needs to reclaim the allocated data storage area, the reference count can be used to determine whether reclamation is possible. Specifically, if the reference count is greater than 0, it indicates that other processes are currently reading the first data in the data storage area, therefore, the reclamation of the data storage area can be postponed. If the reference count is equal to 0, it indicates that no other process is currently reading the first data in the data storage area, therefore, the data storage area can be reclaimed. Therefore, the reliability of program operation can be effectively improved.

[0072] For example, in one embodiment of the present invention, the `export()` and `import()` functions can be implemented in the driver of the target physics accelerator card. The `export()` function marks a region of video memory as shareable, and the `import()` function maintains a reference count variable for this shared video memory region. Each time a second process reads the first data, the reference count of the first data is incremented by 1, and after a second process finishes reading the first data, the reference count is decremented by 1. Thus, when the driver performs video memory reclamation, it can determine whether to reclaim the shared cache based on whether the reference count is 0. Of course, if a region of allocated video memory has not been declared shareable through the `export()` function, it can be directly reclaimed during the reclamation process.

[0073] After implementing the `export()` and `import()` functions in the driver program, the upper-level application framework software can call them to achieve data sharing. For example, in one instance, the upper-level application framework software could be an inter-card data communication library. The program execution method provided in the embodiments of the present invention can modify, extend, and encapsulate this inter-card data communication library, thereby making it compatible and adaptable to higher-level application software.

[0074] For example, in one instance, the inter-card data communication library can be modified to support memory and data sharing between virtual accelerator cards. Specifically, the data acquisition request function `send()` in the original inter-card data communication library can be intercepted, and the `export()` function in the driver can be called within that `send()` function, thereby enabling memory and data sharing. Furthermore, the data reception response function `receive()` in the original inter-card data communication library can be intercepted, and the `import()` function in the driver can be called within that `receive()` function, thereby implementing reference counting for the shared memory, allowing direct use of the shared memory and data, avoiding data copying.

[0075] The following detailed description of the program execution method provided by the embodiments of the present invention is based on a specific example.

[0076] like Figure 5 As shown, the program execution method provided in the embodiments of the present invention may include: S201. Virtualize the physical computing unit of the target physical acceleration card into at least two virtual computing units, the at least two virtual computing units including a first virtual computing unit and a second virtual computing unit; S202. Virtualize the real physical video memory space of the target physics acceleration card into at least two virtual video memories, including a first virtual video memory and a second virtual video memory. Wherein, the first virtual video memory corresponds to the first virtual computing unit, and the second virtual video memory corresponds to the second virtual computing unit; the first virtual video memory and the second virtual video memory uniformly perform mapping from virtual physical video memory space to real physical video memory space; the virtual physical video memory space of the first virtual video memory is mapped to the first region of the real physical video memory space, and the virtual physical video memory space of the second virtual video memory is mapped to the second region of the real physical video memory space, and the first region and the second region are the same; S203. In the driver program of the target physical accelerator card, the first virtual accelerator card is configured to have management permissions for the third area of ​​the real physical video memory space, and the second virtual accelerator card is configured to have management permissions for the fourth area of ​​the real physical video memory space. The third region is different from the fourth region; both the third and fourth regions are located within the first region. S204. Run the first process based on the first virtual accelerator card and obtain the data write request of the first process to write the first data to the real physical video memory space. S205. Based on the data write request, query the address range that the first virtual accelerator card has management authority in the driver of the target physical accelerator card to obtain the third region; S206. Allocate a corresponding data storage area for the data write request in the third area; S207. Write the first data into the data storage area.

[0077] S208. Run a second process based on the second virtual accelerator card, obtain the data read request of the second process, and the data read request carries the target video memory address of the data storage area to be read; S209. According to the data read request, set the data storage area to a shareable state so that the second virtual accelerator card can read the first data from the target video memory address; S210. Increment the reference count of the target video memory address by 1; S211. After the first data reading is completed, the reference count of the target video memory address is decremented by 1.

[0078] S212, Obtain the request to reclaim the data storage area corresponding to the target video memory address; S213. If the reference count of the target memory address is 0, reclaim the data storage area; or, if the reference count of the target memory address is greater than 0, postpone the reclamation of the data storage area.

[0079] Secondly, embodiments of the present invention also provide a program running device that can optimize resource allocation for different processes while saving communication overhead.

[0080] like Figure 6As shown, the program running device provided in the embodiments of the present invention may include an acquisition module 31 and a running module 32. The acquisition module 31 is used to acquire at least two virtual accelerator cards on the same physical accelerator card. The at least two virtual accelerator cards include a first virtual accelerator card and a second virtual accelerator card. The first virtual accelerator card includes a first virtual computing unit and a first virtual video memory, and the second virtual accelerator card includes a second virtual computing unit and a second virtual video memory. The first virtual computing unit and the second virtual computing unit are isolated from each other, and the first virtual video memory and the second virtual video memory share the real physical video memory space mapped to them. The running module 32 is used to run a first process based on the first virtual accelerator card and a second process based on the second virtual accelerator card. The first process and the second process share data through the first virtual video memory and the second virtual video memory during operation.

[0081] The program execution apparatus provided in the embodiments of the present invention can acquire at least two virtual accelerator cards on the same physical accelerator card, run a first process based on the first virtual accelerator card, and run a second process based on the second virtual accelerator card. Since the first virtual computing unit of the first virtual accelerator card and the second virtual computing unit of the second virtual accelerator card are isolated from each other, the computing resources required by the first and second processes can be allocated according to their characteristics, making the allocation of computing resources between different processes more reasonable. Furthermore, since the first and second virtual accelerator cards are located on the same physical accelerator card, the first virtual video memory of the first virtual accelerator card and the second virtual video memory of the second virtual accelerator card share the mapped real physical video memory space. Thus, the first and second processes can obtain the required storage resources and share data based on the shared real physical video memory space without needing to transmit data through time-consuming inter-card communication. Therefore, it can optimize the resource allocation of different processes while saving communication overhead.

[0082] In one embodiment, the acquisition module 31 includes: a first virtualization submodule, used to virtualize the physical computing units of the target physical accelerator card into at least two virtual computing units, the at least two virtual computing units including a first virtual computing unit and a second virtual computing unit, the first virtual computing unit and the second virtual computing unit being established based on different hardware resources on the target physical accelerator card and isolated from each other; a second virtualization submodule, used to virtualize the real physical video memory space of the target physical accelerator card into at least two virtual video memories, the at least two virtual video memories including the first virtual video memory and the second virtual video memory; the first virtual video memory corresponds to the first virtual computing unit, and the second virtual video memory corresponds to the second virtual computing unit; the first virtual video memory and the second virtual video memory are uniformly mapped from virtual physical video memory space to real physical video memory space; the virtual physical video memory space of the first virtual video memory is mapped to a first region of the real physical video memory space, the virtual physical video memory space of the second virtual video memory is mapped to a second region of the real physical video memory space, the first region and the second region are the same; the first process and the second process share data through the first region of the real physical video memory space during operation.

[0083] In one implementation, the first region and the second region are default regions of the real physical video memory space, or the first region and the second region are configured by a virtual mask register in a preset virtual device model.

[0084] In one embodiment, the acquisition module 31 further includes: a setting submodule, configured to, after virtualizing the real physical video memory space of the target physics accelerator card into at least two virtual video memories, in the driver program of the target physics accelerator card, set the first virtual accelerator card to have management authority over a third region of the real physical video memory space, and set the second virtual accelerator card to have management authority over a fourth region of the real physical video memory space, wherein the third region is different from the fourth region; both the third region and the fourth region are located within the first region; the management authority includes allocating video memory space and / or reclaiming video memory space.

[0085] In one implementation, the first virtual video memory and the second virtual video memory are uniformly mapped from virtual physical video memory space to real physical video memory space. The first virtual video memory is visible to a first region of the real physical video memory space, and the second virtual video memory is visible to a second region of the real physical video memory space. The first region and the second region are the same. The first process and the second process share data through the first region of the real physical video memory space during operation.

[0086] In one embodiment, the running module 32 includes: a first running submodule, configured to run the first process based on the first virtual accelerator card, and write the first data generated during the running of the first process into the first region of the real physical video memory space through the first virtual video memory; and a second running submodule, configured to run the second process based on the second virtual accelerator card, and read the first data from the first region of the real physical video memory space through the second virtual video memory for processing by the second process.

[0087] In one implementation, the first virtual accelerator card has management authority over a third region of the real physical video memory space, and the second virtual accelerator card has management authority over a fourth region of the real physical video memory space, wherein the third region is different from the fourth region; both the third region and the fourth region are located within the first region; the management authority includes allocating video memory space and / or reclaiming video memory space; the first running submodule is specifically configured to: run the first process based on the first virtual accelerator card, obtain a data write request from the first process to write the first data to the real physical video memory space; according to the data write request, query the address range in the driver of the physical accelerator card where the first virtual accelerator card has management authority to obtain the third region; allocate a corresponding data storage area for the data write request in the third region; and write the first data to the data storage area.

[0088] In one implementation, the second running submodule is specifically configured to: run the second process based on the second virtual accelerator card; obtain a data read request from the second process, wherein the data read request carries the target video memory address of the data storage area to be read; set the data storage area to a shareable state according to the data read request, so that the second virtual accelerator card reads the first data from the target video memory address; increment the reference count of the target video memory address by 1; and decrement the reference count of the target video memory address by 1 after the first data is read.

[0089] In one implementation, the first running submodule is further configured to: after allocating a corresponding data storage area for the data write request in the third region, obtain a request to reclaim the data storage area corresponding to the target video memory address; reclaim the data storage area when the reference count of the target video memory address is 0; or, postpone the reclamation of the data storage area when the reference count of the target video memory address is greater than 0.

[0090] In one embodiment, the number of the first virtual accelerator cards is at least one, and the number of the second virtual accelerator cards is at least one.

[0091] In one implementation, the first process includes a process executed by the large language model during the pre-filling phase, and the second process includes a process executed by the large language model during the decoding phase.

[0092] Thirdly, embodiments of the present invention also provide an electronic device that can optimize resource allocation for different processes while saving communication overhead.

[0093] like Figure 7 As shown, the electronic device provided in the embodiments of the present invention may include: a processor 91 and a memory 92, wherein the processor 91 and the memory 92 are electrically connected; the memory 92 is used to store executable program code; the processor 91 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 92, so as to implement any of the program running methods provided in the foregoing embodiments.

[0094] For details on the specific execution process of the above steps by the processor 91 and the steps further executed by the processor 91 by running executable program code, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0095] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement any of the program execution methods provided in the foregoing embodiments, thus achieving the corresponding technical effects. This has been described in detail above and will not be repeated here.

[0096] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0097] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0098] In particular, the device embodiment is basically similar to the method embodiment, so the description is relatively simple. For relevant details, please refer to the description of the method embodiment.

[0099] For ease of description, the above apparatus is described by dividing it into various functional units / modules. Of course, in implementing this invention, the functions of each unit / module can be implemented in one or more software and / or hardware.

[0100] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for running a program, characterized in that, include: At least two virtual accelerator cards are obtained on the same physical accelerator card. The at least two virtual accelerator cards include a first virtual accelerator card and a second virtual accelerator card. The first virtual accelerator card includes a first virtual computing unit and a first virtual video memory. The second virtual accelerator card includes a second virtual computing unit and a second virtual video memory. The first virtual computing unit and the second virtual computing unit are isolated from each other. The first virtual video memory and the second virtual video memory share the real physical video memory space mapped to each other. A first process runs based on the first virtual accelerator card, and a second process runs based on the second virtual accelerator card, wherein the first process and the second process share data through the first virtual video memory and the second virtual video memory during operation.

2. The program execution method according to claim 1, characterized in that, The acquisition of at least two virtual accelerator cards on the same physical accelerator card includes: The physical computing unit of the target physical accelerator card is virtualized into at least two virtual computing units, the at least two virtual computing units including the first virtual computing unit and the second virtual computing unit, the first virtual computing unit and the second virtual computing unit are established based on different hardware resources on the target physical accelerator card and are isolated from each other; The real physical memory space of the target physics acceleration card is virtualized into at least two virtual memory locations, including a first virtual memory location and a second virtual memory location. The first virtual memory location corresponds to the first virtual computing unit, and the second virtual memory location corresponds to the second virtual computing unit. The first virtual memory location and the second virtual memory location are uniformly mapped from virtual physical memory space to real physical memory space. The virtual physical memory space of the first virtual memory location is mapped to a first region of the real physical memory space, and the virtual physical memory space of the second virtual memory location is mapped to a second region of the real physical memory space. The first region and the second region are the same. The first process and the second process share data through the first region of the real physical video memory space during operation.

3. The program execution method according to claim 2, characterized in that, The first region and the second region are the default regions of the real physical video memory space, or the first region and the second region are configured by the virtual mask register in the preset virtual device model.

4. The program execution method according to claim 2, characterized in that, After virtualizing the actual physical memory space of the target physics acceleration card into at least two virtual memory locations, the method further includes: In the driver program of the target physics acceleration card, the first virtual acceleration card is configured to have management permissions over a third region of the real physical video memory space, and the second virtual acceleration card is configured to have management permissions over a fourth region of the real physical video memory space. The third region is different from the fourth region. Both the third region and the fourth region are located within the first region. The management permissions include allocating video memory space and / or reclaiming video memory space.

5. The program execution method according to claim 1, characterized in that, The first virtual video memory and the second virtual video memory are uniformly mapped from virtual physical video memory space to real physical video memory space. The first virtual video memory is visible to a first region of the real physical video memory space, and the second virtual video memory is visible to a second region of the real physical video memory space. The first region and the second region are the same. The first process and the second process share data through the first region of the real physical video memory space during operation.

6. The program execution method according to any one of claims 2 to 5, characterized in that, The step of running a first process based on the first virtual accelerator card and running a second process based on the second virtual accelerator card includes: The first process runs based on the first virtual accelerator card, and the first data generated during the running of the first process is written into the first region of the real physical video memory space through the first virtual video memory. The second process runs on the second virtual accelerator card and reads the first data from the first region of the real physical video memory space through the second virtual video memory for processing by the second process.

7. The program execution method according to claim 6, characterized in that, The first virtual accelerator card has management authority over the third region of the real physical video memory space, and the second virtual accelerator card has management authority over the fourth region of the real physical video memory space. The third region is different from the fourth region. Both the third region and the fourth region are located within the first region. The management authority includes allocating video memory space and / or reclaiming video memory space. The step of running the first process based on the first virtual accelerator card and writing the first data generated during the running of the first process into the first region of the real physical video memory space via the first virtual video memory includes: The first process is run based on the first virtual accelerator card, and a data write request is obtained from the first process to write the first data to the real physical video memory space. Based on the data write request, the address range for which the first virtual accelerator card has management authority is queried in the driver of the physical accelerator card to obtain the third region; In the third region, a corresponding data storage area is allocated for the data write request; Write the first data into the data storage area.

8. The program execution method according to claim 7, characterized in that, The step of running the second process based on the second virtual accelerator card and reading the first data from the first region of the real physical video memory space through the second virtual video memory for processing by the second process includes: The second process runs based on the second virtual accelerator card, and the data read request of the second process is obtained. The data read request carries the target video memory address of the data storage area to be read. According to the data read request, the data storage area is set to a shareable state so that the second virtual accelerator card can read the first data from the target video memory address; Increment the reference count of the target memory address by 1; After the first data reading is completed, the reference count of the target memory address is decremented by 1.

9. The program execution method according to claim 8, characterized in that, After allocating a corresponding data storage area for the data write request in the third area, the method further includes: Obtain a request to reclaim the data storage region corresponding to the target video memory address; If the reference count of the target memory address is 0, the data storage area is reclaimed; or, If the reference count of the target memory address is greater than 0, the reclamation of the data storage area is temporarily suspended.

10. The program execution method according to claim 1, characterized in that, The number of the first virtual accelerator cards is at least one, and the number of the second virtual accelerator cards is at least one.

11. The program execution method according to claim 1, characterized in that, The first process includes the process executed by the large language model in the pre-filling stage, and the second process includes the process executed by the large language model in the decoding stage.

12. A program execution device, characterized in that, include: The acquisition module is used to acquire at least two virtual accelerator cards on the same physical accelerator card. The at least two virtual accelerator cards include a first virtual accelerator card and a second virtual accelerator card. The first virtual accelerator card includes a first virtual computing unit and a first virtual video memory. The second virtual accelerator card includes a second virtual computing unit and a second virtual video memory. The first virtual computing unit and the second virtual computing unit are isolated from each other. The first virtual video memory and the second virtual video memory share the real physical video memory space mapped to each other. The running module is used to run a first process based on the first virtual accelerator card and a second process based on the second virtual accelerator card, wherein the first process and the second process share data through the first virtual video memory and the second virtual video memory during operation.

13. The program execution device according to claim 12, characterized in that, The acquisition module includes: The first virtualization submodule is used to virtualize the physical computing unit of the target physical accelerator card into at least two virtual computing units. The at least two virtual computing units include the first virtual computing unit and the second virtual computing unit. The first virtual computing unit and the second virtual computing unit are established based on different hardware resources on the target physical accelerator card and are isolated from each other. The second virtualization submodule is used to virtualize the real physical memory space of the target physical accelerator card into at least two virtual memory locations, the at least two virtual memory locations including the first virtual memory location and the second virtual memory location; the first virtual memory location corresponds to the first virtual computing unit, and the second virtual memory location corresponds to the second virtual computing unit; the first virtual memory location and the second virtual memory location are uniformly mapped from virtual physical memory space to real physical memory space; the virtual physical memory space of the first virtual memory location is mapped to a first region of the real physical memory space, and the virtual physical memory space of the second virtual memory location is mapped to a second region of the real physical memory space, the first region and the second region are the same; The first process and the second process share data through the first region of the real physical video memory space during operation.

14. The program execution device according to claim 13, characterized in that, The first region and the second region are the default regions of the real physical video memory space, or the first region and the second region are configured by the virtual mask register in the preset virtual device model.

15. The program execution device according to claim 13, characterized in that, The acquisition module further includes: A submodule is configured to, after virtualizing the real physical video memory space of the target physics acceleration card into at least two virtual video memory locations, configure the first virtual accelerator card in the driver program of the target physics acceleration card to have management permissions for a third region of the real physical video memory space, and configure the second virtual accelerator card to have management permissions for a fourth region of the real physical video memory space, wherein the third region is different from the fourth region; both the third region and the fourth region are located within the first region; the management permissions include allocating video memory space and / or reclaiming video memory space.

16. The program execution device according to claim 12, characterized in that, The first virtual video memory and the second virtual video memory are uniformly mapped from virtual physical video memory space to real physical video memory space. The first virtual video memory is visible to a first region of the real physical video memory space, and the second virtual video memory is visible to a second region of the real physical video memory space. The first region and the second region are the same. The first process and the second process share data through the first region of the real physical video memory space during operation.

17. The program execution apparatus according to any one of claims 13 to 16, characterized in that, The operating module includes: The first running submodule is used to run the first process based on the first virtual accelerator card, and write the first data generated during the running of the first process into the first area of ​​the real physical video memory space through the first virtual video memory. The second running submodule is used to run the second process based on the second virtual accelerator card, and to read the first data from the first area of ​​the real physical video memory space through the second virtual video memory for the second process to process.

18. The program execution device according to claim 17, characterized in that, The first virtual accelerator card has management authority over the third region of the real physical video memory space, and the second virtual accelerator card has management authority over the fourth region of the real physical video memory space. The third region is different from the fourth region. Both the third region and the fourth region are located within the first region. The management authority includes allocating video memory space and / or reclaiming video memory space. The first running submodule is specifically used for: The first process is run based on the first virtual accelerator card, and a data write request is obtained from the first process to write the first data to the real physical video memory space. Based on the data write request, the address range for which the first virtual accelerator card has management authority is queried in the driver of the physical accelerator card to obtain the third region; In the third region, a corresponding data storage area is allocated for the data write request; Write the first data into the data storage area.

19. The program execution device according to claim 18, characterized in that, The second running submodule is specifically used for: The second process runs based on the second virtual accelerator card, and the data read request of the second process is obtained. The data read request carries the target video memory address of the data storage area to be read. According to the data read request, the data storage area is set to a shareable state so that the second virtual accelerator card can read the first data from the target video memory address; Increment the reference count of the target memory address by 1; After the first data reading is completed, the reference count of the target memory address is decremented by 1.

20. The program execution device according to claim 19, characterized in that, The first running submodule is also used for: After allocating a corresponding data storage area for the data write request in the third region, a request to reclaim the data storage area corresponding to the target memory address is obtained; If the reference count of the target memory address is 0, the data storage area is reclaimed; or, if the reference count of the target memory address is greater than 0, the reclamation of the data storage area is temporarily suspended.

21. The program execution device according to claim 12, characterized in that, The number of the first virtual accelerator cards is at least one, and the number of the second virtual accelerator cards is at least one.

22. The program execution device according to claim 12, characterized in that, The first process includes the process executed by the large language model in the pre-filling stage, and the second process includes the process executed by the large language model in the decoding stage.

23. An electronic device, characterized in that, The electronic device includes: a processor and a memory, wherein the processor is electrically connected to the memory; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the program running method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the program execution method according to any one of claims 1 to 11.

Images