Model training method and device based on extended display memory, model inference method and device, medium and product

Abstract

The present disclosure provides an extended video memory-based model training method, a model inference method, an apparatus, a device, a medium and a product, relating to the technical field of storage devices. The extended video memory-based model training method comprises: sending a first DSM instruction to an SSD controller, and performing multiple rounds of iterative training by a GPU. Each round of iterative training comprises multiple processing stages. The processing stages comprise: the SSD controller loading data units before calculation corresponding to hierarchical identifiers to the GPU, and the GPU calculating calculation units based on the data units. The first DSM instruction marks the priority of the calculation units. The GPU unloads the calculation units to the SSD. The SSD controller unloads the calculation units of high priority to SLC and unloads the calculation units of low priority to QLC. Through the technical solution of the present disclosure, the access delay of high-sensitive data is reduced without increasing the hardware cost, and the invalid erasing and writing of SLC is reduced to prolong the service life of the SSD.

Description

Technical Field

[0001] This disclosure relates to the field of storage device technology, and in particular to a model training method based on extended video memory, a model inference method based on extended video memory, a model training device based on extended video memory, a model inference device based on extended video memory, an electronic device, a storage device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Because GPUs (Graphics Processing Units) have advantages over CPUs (Central Processing Units) in terms of parallel processing capabilities, throughput, and optimization for machine learning tasks, they have become the preferred choice for training complex and large-scale machine learning models. However, with the development of artificial intelligence technology, the consumption of GPU memory during the training of big data models with billions or more parameters has increased dramatically. The onboard memory capacity of a single GPU is no longer sufficient to meet its storage needs. Therefore, there is an urgent need for a technical solution that uses SSDs (Solid State Drives) to expand the memory to achieve model training and inference.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide a model training method, model inference method, apparatus, device, medium and product based on extended video memory, which at least to some extent overcomes the problem in the related art that the onboard video memory capacity of a single GPU is insufficient to meet the storage requirements of model training and inference.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to one aspect of this disclosure, a model training method based on extended video memory is provided, applied to a host computer, the host computer being connected to a graphics processing unit (GPU) and a solid-state drive (SSD) serving as the extended video memory, the SSD including a high-speed storage area (SLC) and a normal storage area (QLC), the GPU being used to perform iterative training of the big data model, comprising: dividing the big data model to be trained into multiple network layer hierarchical identifiers at the network layer granularity, and storing the data units in the SSD; establishing a DSM instruction interaction channel with the SSD controller, the instruction interaction channel being used to issue DSM instructions of the data access attribute hint type to the SSD; during the multi-round iterative training process performed by the GPU, based on The instruction interaction channel sends a first DSM instruction to the SSD controller. Each round of iterative training includes multiple processing stages. Each processing stage includes: the SSD controller loading the data unit corresponding to the layer identifier before computation onto the GPU, and the GPU calculating the computation unit based on the data unit. The first DSM instruction is used to mark the priority of the computation unit. The processing stage also includes: the GPU unloading the computation unit from the SSD, so that the SSD controller unloads the computation unit with high priority to the SLC and the computation unit with low priority to the QLC. The marked computation unit is configured as the data unit to be loaded in the next processing stage.

[0007] In some embodiments of this disclosure, the multiple processing stages include a forward propagation stage in which the multiple network layers sequentially perform forward computation. The corresponding data unit is a first data unit including hierarchical model parameters. Before the SSD controller loads the pre-computation data unit corresponding to the hierarchical identifier to the GPU, the process further includes: sending a second DSM instruction to the SSD controller based on the instruction interaction channel. The second DSM instruction is used to instruct the SSD controller to directly load the corresponding first data unit from the SLC to the GPU based on the hierarchical identifier of the layer to be processed, or to first migrate the first data unit from the QLC to the SLC and then load it to the GPU, so that the GPU performs forward computation based on the hierarchical model parameters in the direction from input to output to generate hierarchical activation values, and uses the hierarchical model parameters and the corresponding hierarchical activation values ​​as the first computation unit corresponding to the forward propagation stage.

[0008] In some embodiments of this disclosure, before sending the first DSM instruction to the SSD controller based on the instruction interaction channel, the process includes: receiving first information sent by the GPU, wherein the first information is sent by the GPU when it detects that the first computing unit's memory usage reaches a preset threshold, thereby triggering the host to send the corresponding first DSM instruction, wherein the corresponding first DSM instruction is used to determine the first computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as high priority, and to determine the first computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as low priority.

[0009] In some embodiments of this disclosure, the multiple processing stages further include a backpropagation stage in which the multiple network layers perform reverse computation layer by layer. The corresponding data unit is a second data unit including hierarchical model parameters and hierarchical activation values. Before sending the first DSM instruction to the SSD controller based on the instruction interaction channel, the process includes: receiving second information sent by the GPU. The second information is sent by the GPU before unloading the second computing unit to trigger the host to send the corresponding first DSM instruction. The corresponding first DSM instruction is used to determine the second computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as high priority, and to determine the second computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as low priority. The second computing unit includes gradient values ​​generated by the GPU performing reverse computation based on the hierarchical model parameters and the hierarchical activation values ​​in the direction from output to input.

[0010] In some embodiments of this disclosure, the plurality of processing stages further include an optimizer update stage for updating the hierarchical model parameters of each network layer. Before the SSD controller loads the pre-computation data unit corresponding to the hierarchical identifier to the GPU, the process further includes: sending a third DSM instruction to the SSD controller based on the instruction interaction channel. The third DSM instruction is used to instruct the SSD controller to migrate the hierarchical model parameters and optimizer state to the SLC where the gradient value with the same hierarchical identifier is located. The hierarchical model parameters, the optimizer state, and the gradient value form a third data unit.

[0011] In some embodiments of this disclosure, before sending a first DSM instruction to the SSD controller based on the instruction interaction channel, the process includes: receiving third information sent by the GPU, the third information being sent by the GPU before the overall unloading of the third computing unit obtained based on the third data unit, to trigger the host to send the corresponding first DSM instruction. The corresponding first DSM instruction is used to determine the third computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as high priority, and to determine the third computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as low priority. The GPU reads the third data unit from the SLC and performs an update of the hierarchical model parameters and the optimizer state based on the third data unit in the direction from input to output, so as to use the updated hierarchical model parameters and the optimizer state as the third computing unit.

[0012] In some embodiments of this disclosure, the big data model to be trained is hierarchically divided into multiple network layer hierarchical identifiers at the network layer granularity, including: reading the network structure configuration information of the big data model to be trained, and determining multiple network layers and corresponding hierarchical identifiers based on the reading results, wherein, based on the processing stage, at least one of the model parameters, activation values, gradients and optimizer states corresponding to each hierarchical identifier is determined as the data unit.

[0013] In some embodiments of this disclosure, establishing a DSM command interaction channel with an SSD controller includes: initializing the communication interface with the SSD controller to configure the transmission protocol, data format, and interaction timing of the DSM commands; and completing a handshake authentication with the SSD controller based on the transmission protocol, the data format, and the interaction timing to establish the command interaction channel.

[0014] According to another aspect of this disclosure, a model training method based on extended video memory is provided, applied to an SSD controller that serves as the extended video memory. The SSD controller is connected to both a GPU and a host. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC). The method includes: establishing a DSM instruction interaction channel with the host, the instruction interaction channel being used to receive DSM instructions of the data access attribute hint type; loading pre-computation data units onto the GPU so that the GPU can compute computation units based on the data units, the data units corresponding to the layer identifiers of the network layers to be trained in the big data model; receiving a first DSM instruction sent by the host based on the instruction interaction channel; marking the priority of the computation units based on the first DSM instruction, and unloading the computation units with high priority to the SLC and the computation units with low priority to the QLC to complete a processing stage, wherein the marked computation units are configured as the data units to be loaded in the next processing stage, and the GPU performs multiple rounds of iterative training, each round of iterative training including multiple processing stages.

[0015] In some embodiments of this disclosure, multiple processing stages include a forward propagation stage, and the corresponding data unit is a first data unit including hierarchical model parameters. Loading the data unit before computation onto the GPU includes: receiving a second DSM instruction sent by the host based on the instruction interaction channel; based on the second DSM instruction and the hierarchical identifier of the layer to be processed, directly loading the corresponding first data unit from the SLC to the GPU, or first migrating the first data unit from the QLC to the SLC and then loading it onto the GPU, so that the GPU performs forward computation based on the hierarchical model parameters in the direction from input to output to generate hierarchical activation values, and using the hierarchical model parameters and the corresponding hierarchical activation values ​​as the corresponding first computation unit; marking the priority of the computation unit based on the first DSM instruction includes: determining the first computation unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as high priority based on the first DSM instruction, and determining the first computation unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as low priority.

[0016] In some embodiments of this disclosure, multiple processing stages further include a backpropagation stage, where the corresponding data unit is a second data unit including hierarchical model parameters and hierarchical activation values. Loading the pre-computation data unit onto the GPU includes: directly transmitting the corresponding second data unit from the SLC to the GPU based on the hierarchical identifier of the layer to be processed, or migrating the second data unit from the QLC to the SLC first, and then transmitting it to the GPU, so that the GPU performs backpropagation based on the hierarchical model parameters and the hierarchical activation values ​​in the direction from output to input to generate gradient values, and using the gradient values ​​as the corresponding second computation unit; prioritizing the computation unit based on the first DSM instruction includes: determining the second computation unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as high priority, and determining the second computation unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as low priority based on the first DSM instruction.

[0017] In some embodiments of this disclosure, multiple processing stages further include an optimizer update stage for updating the hierarchical model parameters of each network layer, and loading pre-computation data units onto the GPU, including: receiving a third DSM instruction sent by the host based on the instruction interaction channel; migrating the hierarchical model parameters and optimizer state to the SLC containing gradient values ​​with the same hierarchical identifier based on the third DSM instruction, wherein the hierarchical model parameters, the optimizer state, and the gradient values ​​form a third data unit, so that the GPU generates a third computation unit based on the third data unit; and marking the priority of the computation unit based on the first DSM instruction, including: determining the third computation unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as high priority, and determining the third computation unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as low priority.

[0018] In some embodiments of this disclosure, establishing a DSM command interaction channel with the host includes: responding to an initialization configuration operation of the communication interface initiated by the host, obtaining the transmission protocol, data format, and interaction timing of the DSM command; and completing a handshake authentication with the host based on the transmission protocol, the data format, and the interaction timing to establish the command interaction channel.

[0019] According to another aspect of this disclosure, a model inference method based on extended video memory is provided, applied to a host computer connected to a graphics processing unit (GPU) and a solid-state drive (SSD) serving as the extended video memory. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC). The method includes: hierarchically dividing a big data model at the network layer level to obtain model layers, each model layer including hierarchical weights, the usage frequency of which increases from the input to the output of the big data model; establishing a DSM (Data Access Module) command interaction channel with the SSD controller, the command interaction channel being used to issue DSM commands of the data access attribute hint type to the SSD; and responding to the inference startup operation of the big data model, based on... The system sends a second DSM instruction to the SSD controller via the instruction interaction channel. The second DSM instruction is used to load a first set of weights into the SLC and a second set of weights into the QLC. The first set of weights includes the hierarchical weights that are greater than the weight threshold, and the second set of weights includes the hierarchical weights that are less than or equal to the weight threshold. In response to an acquired inference request, the GPU is triggered to perform inference of the big data model based on the hierarchical weights to generate a KV cache. The GPU also sends a DSM instruction corresponding to the type of the inference request to the SSD controller via the instruction interaction channel, so that the SSD writes the KV cache into the SLC and / or QLC based on the corresponding DSM instruction.

[0020] In some embodiments of this disclosure, sending a DSM instruction corresponding to the type of the inference request to the SSD controller based on the instruction interaction channel includes: identifying the inference request as a high-priority request and sending a corresponding first DSM instruction to the SSD controller, wherein the first DSM instruction is used to configure a high-priority storage policy for the KV cache to write to the SLC; and identifying the inference request as a low-priority request and sending a corresponding first DSM instruction to the SSD controller, wherein the first DSM instruction is used to configure a low-priority storage policy for the KV cache to write to the QLC.

[0021] In some embodiments of this disclosure, sending a DSM instruction corresponding to the type of the inference request to the SSD controller based on the instruction interaction channel includes: in response to the inference request, identifying the inference request as an active request, and sending a corresponding third DSM instruction to the SSD controller, the third DSM instruction being used to write the generated KV cache to the SLC; identifying the inference request as an idle request, and sending a corresponding third DSM instruction to the SSD controller, the third DSM instruction being used to downgrade the generated historical KV cache to the QLC.

[0022] In some embodiments of this disclosure, identifying the inference request as an idle request further includes: sending a fourth DSM instruction to the SSD controller based on the instruction interaction channel, the fourth DSM instruction being used to instruct the SSD to release the degraded historical KV cache.

[0023] In some embodiments of this disclosure, the method further includes: in response to an inference end prompt from the GPU, sending a fourth DSM instruction to the SSD controller, the fourth DSM instruction causing the SSD to release the generated KV cache.

[0024] According to another aspect of this disclosure, a model inference method based on extended video memory is provided, applied to an SSD controller serving as the extended video memory. The SSD controller is connected to both a GPU and a host. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC). The method includes: establishing a DSM instruction interaction channel with the host, the instruction interaction channel being used to receive DSM instructions of the data access attribute hint type; receiving a second DSM instruction sent by the host based on the instruction interaction channel; loading a first set of weights into the SLC and a second set of weights into the QLC based on the second DSM instruction, wherein the first set of weights includes hierarchical weights greater than a weight threshold, and the second set of weights includes hierarchical weights less than or equal to the weight threshold, the hierarchical weights being the weights of model layers obtained by hierarchically dividing the big data model at the network layer granularity; receiving a DSM instruction sent by the host based on the instruction interaction channel corresponding to the type of inference request of the big data model, and the GPU executing the inference operation of the big data model to generate a key-value cache.

[0025] In some embodiments of this disclosure, receiving a DSM instruction sent by the host based on the instruction interaction channel, corresponding to the type of inference request of the big data model, includes: receiving a first DSM instruction; configuring a high-priority storage strategy for the KV cache based on the first DSM instruction to write to the SLC; or configuring a low-priority storage strategy for the KV cache to be generated to write to the QLC.

[0026] In some embodiments of this disclosure, receiving a DSM instruction sent by the host based on the instruction interaction channel, corresponding to the type of inference request of the big data model, includes: receiving a third DSM instruction; using the third DSM instruction to write the generated KV cache into the SLC; or using it to downgrade the generated historical KV cache into the QLC.

[0027] In some embodiments of this disclosure, the method further includes: receiving a fourth DSM instruction sent by the host based on the instruction interaction channel; and releasing the degraded historical KV cache based on the fourth DSM instruction.

[0028] In some embodiments of this disclosure, the method further includes: receiving a fourth DSM instruction; and releasing the KV cache in the SLC based on the fourth DSM instruction.

[0029] According to another aspect of this disclosure, a model training device based on extended video memory is provided, applied to a host computer, wherein the host computer is connected to a graphics processing unit (GPU) and a solid-state drive (SSD) serving as the extended video memory, the SSD comprising a high-speed storage area (SLC) and a normal storage area (QLC), comprising: a first partitioning unit, configured to partition the large data model to be trained hierarchically at the network layer level to obtain multiple network layer hierarchical identifiers, and store the hierarchical identifiers in the SSD; a first establishment unit, configured to establish a DSM (Data Access Attribute) command interaction channel with the SSD controller, the command interaction channel being used to send DSM commands of the data access attribute hint type to the SSD; and a first sending unit, configured to send commands to the SSD based on the command interaction channel. The controller sends a first DSM instruction, wherein the GPU performs multiple rounds of iterative training, each round of iterative training including multiple processing stages, the processing stage including: the SSD controller loading the data unit corresponding to the level identifier to the GPU before computation and the GPU calculating the computation unit based on the data unit, the first DSM instruction being used to mark the priority of the computation unit, the processing stage further including: the GPU unloading the computation unit to the SSD, so that the SSD controller unloads the computation unit with high priority to the SLC and the computation unit with low priority to the QLC, the marked computation unit being configured as the data unit to be loaded in the next processing stage.

[0030] According to another aspect of this disclosure, a model training apparatus based on extended video memory is provided, applied to an SSD controller serving as the extended video memory. The SSD controller is connected to both a GPU and a host. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC). The apparatus includes: a second establishment unit for establishing a DSM instruction interaction channel with the host, the instruction interaction channel being used to receive DSM instructions of the data access attribute hint type; a first loading unit for loading pre-computation data units onto the GPU so that the GPU can compute computation units based on the data units, the data units corresponding to the layer identifiers of the network layers to be trained in the big data model; a first receiving unit for receiving a first DSM instruction sent by the host based on the instruction interaction channel; and a marking unit for marking the priority of the computation units based on the first DSM instruction, unloading high-priority computation units to the SLC, and unloading low-priority computation units to the QLC to complete a processing stage. The marked computation units are configured as data units to be loaded in the next processing stage. The GPU performs multiple rounds of iterative training, each round of iterative training including multiple processing stages.

[0031] According to another aspect of this disclosure, a model inference device based on extended video memory is provided, applied to a host computer, the host computer being connected to a graphics processing unit (GPU) and a solid-state drive (SSD) serving as the extended video memory, the SSD including a high-speed storage area (SLC) and a normal storage area (QLC), comprising: a second partitioning unit, used to partition a big data model hierarchically at the network layer level to obtain model layers, the model layers including hierarchical weights, the usage frequency of the hierarchical weights increasing from the input to the output of the big data model; a third establishment unit, used to establish a DSM command interaction channel with the SSD controller, the command interaction channel being used to send DSM commands of the data access attribute hint type to the SSD; and a second sending unit, used to respond to the big data model. The inference initiation operation involves sending a second DSM instruction to the SSD controller via the instruction interaction channel. The second DSM instruction is used to load a first set of weights into the SLC and a second set of weights into the QLC. The first set of weights includes the hierarchical weights that are greater than a weight threshold, and the second set of weights includes the hierarchical weights that are less than or equal to the weight threshold. A third sending unit is used to, in response to the acquired inference request, trigger the GPU to perform inference of the big data model based on the hierarchical weights to generate a KV cache, and send a DSM instruction corresponding to the type of the inference request to the SSD controller via the instruction interaction channel, so that the SSD writes the KV cache into the SLC and / or QLC based on the corresponding DSM instruction.

[0032] According to another aspect of this disclosure, a model inference apparatus based on extended video memory is provided, applied to an SSD controller serving as the extended video memory. The SSD controller is connected to both a GPU and a host. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC). The apparatus includes: a fourth establishment unit for establishing a DSM instruction interaction channel with the host, the instruction interaction channel being used to receive DSM instructions of the data access attribute hint type; a second receiving unit for receiving a second DSM instruction sent by the host based on the instruction interaction channel; and a second loading unit for loading a first set of weights based on the second DSM instruction. The SLC loads a second set of weights into the QLC. The first set of weights includes hierarchical weights greater than a weight threshold, and the second set of weights includes hierarchical weights less than or equal to the weight threshold. The hierarchical weights are the weights of the model layers obtained by hierarchically dividing the big data model at the network layer level. The third receiving unit is used to receive DSM instructions sent by the host based on the instruction interaction channel, corresponding to the type of inference request of the big data model. The GPU executes the inference operation of the big data model to generate a KV cache. The writing unit is used to write the generated KV cache into the SLC and / or QLC based on the corresponding DSM instructions.

[0033] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the extended memory-based model training method or the extended memory-based model inference method described above by executing the executable instructions.

[0034] According to another aspect of this disclosure, a storage device is provided, comprising: a controller; an SLC electrically connected to the controller; and a QLC electrically connected to the controller, wherein the controller is configured to execute the extended memory-based model training method or the extended memory-based model inference method described in any one of the preceding claims.

[0035] According to another aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the extended video memory-based model training method or the extended video memory-based model inference method of any of the above.

[0036] According to another aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the extended video memory-based model training method or the extended video memory-based model inference method of any of the above.

[0037] The model training scheme and inference method based on extended GPU memory provided in the embodiments of this disclosure divide training data at the network layer level and combine the access characteristics of each stage of big data model training. After the GPU executes the big data model training to obtain computing units, the first DSM instruction is used to mark the access priority of the computing units. This causes the SSD controller to offload high-priority computing units to the SLC cache and store low-priority computing units in the QLC storage area. This achieves the matching of data access requirements with storage medium performance. Without increasing hardware costs, it can reduce the access latency of highly sensitive data, reduce invalid SLC erases and writes to improve the utilization efficiency of SLC cache resources, and prevent high-frequency read and write aging of QLC to extend the life of SSD. Thus, while expanding the GPU memory capacity, it is beneficial to improve the overall training efficiency of big data model training.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0040] Figure 1 This diagram illustrates a computer system according to an embodiment of the present disclosure; Figure 2 A flowchart illustrating the host executing a model training method based on extended video memory in an embodiment of this disclosure is shown; Figure 3 This diagram illustrates the data interaction between the GPU and SSD during the forward propagation phase of model training in an embodiment of this disclosure. Figure 4 This diagram illustrates the data interaction between the GPU and SSD during the backpropagation phase of model training in an embodiment of this disclosure. Figure 5 This diagram illustrates the data interaction between the GPU and SSD during the optimizer update phase of model training in an embodiment of this disclosure. Figure 6 A flowchart illustrating the SSD performing a model training method based on extended video memory in an embodiment of this disclosure is shown. Figure 7 A flowchart illustrating the host executing a model inference method based on extended video memory in an embodiment of this disclosure is shown; Figure 8A flowchart illustrating the SSD performing a model inference method based on extended video memory in an embodiment of this disclosure is shown; Figure 9 A flowchart illustrating a model inference method based on the interaction between the host, GPU, and SSD controller in an embodiment of this disclosure is shown. Figure 10 A schematic diagram a of a first model training device based on extended video memory is shown in an embodiment of this disclosure; Figure 11 A schematic diagram b of the second model training device based on extended video memory is shown in an embodiment of this disclosure; Figure 12 A schematic diagram a of a first model inference device based on extended video memory is shown in an embodiment of this disclosure; Figure 13 A schematic diagram b of a second model inference device based on extended video memory is shown in an embodiment of this disclosure. Detailed Implementation

[0041] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0042] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0043] In the system architecture, the host is a CPU-centric control system responsible for overall task scheduling, instruction issuance, and resource management, acting as the master control unit in big data model training or inference. The GPU is the acceleration core focused on massively parallel computing, primarily undertaking intensive computational tasks such as model forward propagation and backward propagation. The two interact and collaborate via a high-speed bus, forming a master-slave architecture where the host handles control and policy scheduling, and the GPU handles the actual computational execution. With the rapid development of artificial intelligence technology, large language models with billions of parameters or more have become the mainstream application, and the demand for GPU memory during their training and inference processes continues to rise. Limited by hardware costs and physical specifications, the memory capacity of a single GPU is insufficient to meet the high-speed storage requirements of all data, including parameters, activation values, and gradients, in big data models. Therefore, using SSDs as extended memory has become an effective technical means to reduce the hardware threshold for deploying big data models and improve the utilization rate of single-GPU computing power.

[0044] QLC (Quad-Level Cell) SSDs (Solid-State Drives) with SLC (Single-Level Cell) caching employ a hybrid architecture combining high-performance caching and large-capacity storage. They utilize high-capacity, low-cost QLC flash memory as the primary persistent storage medium, and dynamically allocate a portion of the flash memory space through firmware to simulate SLC mode and build a high-speed cache. During data read / write operations, the SLC cache is accessed first to achieve high bandwidth and low latency performance. When the cached data reaches a certain amount or the system is idle, it is migrated to the QLC area for storage by the controller in the background. This approach retains the advantages of QLC's large capacity and low cost while mitigating the weaknesses of native QLC's write performance and large latency fluctuations, achieving a comprehensive balance between storage capacity, read / write performance, and hardware cost.

[0045] NVMe (Non-Volatile Memory Express) is a high-performance transmission protocol designed specifically for flash memory devices such as SSDs. It enables high-speed interconnection between the host and storage devices via PCIe lines. The NVMe DSM (Dataset Management) mechanism allows the host to send data access attribute prompts to the SSD controller, dynamically scheduling data to the SLC cache or QLC storage area. This ensures that high-priority, low-latency sensitive data is matched with high-performance SLC media, improving the system's effective read and write performance without significantly increasing hardware costs, while also mitigating flash memory wear and extending SSD lifespan.

[0046] The following will describe in more detail each step of the model training method based on extended video memory in this exemplary embodiment, with reference to the accompanying drawings and embodiments.

[0047] like Figure 1 As shown, the architecture of the big data model training / inference system disclosed herein includes: a host, a GPU, and a QLC SSD with SLC cache, which are interconnected via a PCIe high-speed bus. The host performs task scheduling and state management of the GPU through the master control instruction channel, and simultaneously issues data attribute configuration instructions to the SSD controller through the DSM instruction channel. The GPU integrates a computing core and VRAM for performing parallel computations related to the model and temporarily storing hot data. The GPU and SSD communicate and transmit parameters, activation values, cache, and other data through a high-speed data read / write channel. Under the scheduling of the controller, the SSD uses the SLC cache to process high-priority, low-latency data and uses the QLC area to achieve large-capacity persistent data storage, together forming a complete system architecture based on SSD extended video memory.

[0048] Figure 2 The flowchart of a model training method based on extended video memory is shown in an embodiment of this disclosure.

[0049] like Figure 2 As shown, the host computer executes a model training method based on extended GPU memory, including the following steps: Step S202: The big data model to be trained is divided into layers at the granularity of network layers to obtain layer identifiers for multiple network layers, and the layer identifiers are stored in SSD.

[0050] The complete training process of a big data model can include three processing stages: forward propagation, back propagation, and optimizer update. In each processing stage, the data unit is the data before the execution of data computation.

[0051] In some embodiments, a big data model can be understood as a deep learning model with a parameter scale of one billion or more and a multi-layer network structure. The network layer is the basic computing unit that constitutes the model, and different layers undertake different functions such as feature extraction, transformation and mapping. The data unit is the smallest set of data that is loaded and computed at each stage of training, divided according to the granularity of the network layer. It may include the model parameters, activation values, gradients and optimizer states corresponding to that layer.

[0052] Step S204: Establish a DSM command interaction channel with the SSD controller. The command interaction channel is used to send DSM commands of the data access attribute prompt type to the SSD.

[0053] In some embodiments, establishing a DSM command interaction channel enables the host to finely schedule SSD storage behavior, allowing the SSD to perceive data access priority, access mode and life cycle in advance, thereby dynamically scheduling high-priority data to SLC cache and storing low-priority data in QLC storage area, which not only improves the access speed of critical data, but also prevents SLC resource waste, and balances flash memory wear.

[0054] Step S206: Send a first DSM instruction to the SSD controller via the interaction channel. The GPU performs multiple rounds of iterative training. Each round of iterative training includes multiple processing stages. The processing stages include: the SSD controller loads the data units corresponding to the hierarchical identifiers to the GPU before computation and the GPU calculates the computation units based on the data units. The first DSM instruction is used to mark the priority of the computation units. The processing stage also includes: the GPU unloads the computation units to the SSD so that the SSD controller unloads the computation units with high priority to the SLC and unloads the computation units with low priority to the QLC. The marked computation units are configured as data units to be loaded in the next processing stage.

[0055] In some embodiments, each round of iterative training includes multiple processing stages such as forward propagation, back propagation, and optimizer update. For each network layer obtained by dividing the big data model, these multiple processing stages will be executed in sequence. That is, each network layer participates in the computation and data scheduling of all processing stages in a single iteration, thereby realizing full-process iterative training based on hierarchical granularity.

[0056] In some embodiments, each round of iterative training includes multiple processing stages. The first DSM instruction can be sent to each processing stage within a single round of iteration. That is, the number of times the first DSM instruction is sent corresponds to the number of processing stages in the round of iteration. For example, if a round of iteration includes three processing stages, the first DSM instruction is sent only three times in that round of iteration. Alternatively, the first DSM instruction can be sent only once, and different processing stages are all executed based on the same first DSM instruction.

[0057] In some embodiments, the multi-round iterations performed by the GPU constitute the execution flow of big data model training. Each round of iteration fully covers three consecutive processing stages: forward propagation, back propagation, and optimizer update. Specifically, the GPU first loads the data units of each layer in forward order to perform forward computation, generating and temporarily storing computational units such as activation values. Then, it loads the activation values ​​from the forward stage in reverse order to perform back propagation, generating computational units such as gradients. Finally, it loads the parameters and gradients in forward order to perform optimizer updates, completing parameter iterative optimization. The multi-round iterations repeat the above complete process until the model converges or reaches the preset number of training rounds. During this process, the host continuously schedules the storage of data in the SLC and QLC intervals of each stage through DSM instructions to ensure the computational efficiency of each round of iteration and the effective utilization of storage resources.

[0058] In some embodiments, the first DSM instruction is a DSM Dataset Range instruction, which is used to mark access priority, access frequency and lifecycle attributes for a specified data range (such as computing units divided by network layer), so that the SSD controller can schedule high-priority, low-latency sensitive data to the SLC cache area and low-priority, high-capacity data to the QLC storage area, thereby improving system read and write performance and balancing flash memory wear without increasing hardware costs.

[0059] In some embodiments, a computation unit is a set of output data generated by the GPU after performing forward, backward, or optimizer computations based on the input data units. Examples include activation values ​​obtained from forward propagation, gradients obtained from backward propagation, and parameters updated by the optimizer. This set of data is both the computation result of the current stage and the input data for the next stage.

[0060] In this embodiment, by dividing the training data at the network layer level and combining the access characteristics of each stage of big data model training, after the GPU executes the big data model training to obtain computing units, the first DSM instruction is used to mark the access priority of the computing units. This causes the SSD controller to offload high-priority computing units to the SLC cache and store low-priority computing units in the QLC storage area. This achieves the matching of data access requirements with storage medium performance. Without increasing hardware costs, it can reduce the access latency of highly sensitive data, reduce invalid SLC erases and writes to improve the utilization efficiency of SLC cache resources, and prevent high-frequency read and write aging of QLC to extend the life of SSD. Thus, while expanding the GPU memory capacity, it is beneficial to improve the overall training efficiency of big data model training.

[0061] In some embodiments of this disclosure, multiple processing stages include a forward propagation stage in which multiple network layers perform forward computation sequentially. The corresponding data unit is a first data unit including hierarchical model parameters. Before sending the first DSM instruction to the SSD controller based on the interaction channel, the process includes: receiving first information sent by the GPU. The first information is sent by the GPU when it detects that the memory occupancy rate of the first computing unit reaches a preset threshold, thereby triggering the host to send the corresponding first DSM instruction. The corresponding first DSM instruction is used to determine the first computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as high priority, and to determine the first computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as low priority.

[0062] The designated position can be understood as a pre-defined position in the big data model, used to distinguish the access popularity of the preceding and following levels, located between two intermediate levels in the big data model. The specific positions of the two intermediate levels can be determined according to the volume of the SLC. If the volume of the SLC is large, the number of high-priority levels can be more than the number of low-priority level identifiers. If the volume of the SLC is small, the number of high-priority levels can be less than or equal to the number of low-priority level identifiers.

[0063] In some embodiments, the corresponding first DSM instruction here refers to a first DSM instruction adapted to the forward propagation phase for prioritizing the first computational units generated in that phase.

[0064] In some embodiments, the first data unit is the input data that the GPU needs to load before performing computation in the forward propagation phase. Specifically, it is the hierarchical model parameters corresponding to each network layer, which is the basic input for forward computation. The first information is the status trigger information sent by the GPU to the host. It is sent when the GPU detects that the current first computation unit's memory usage has reached a preset threshold and the data needs to be unloaded to the SSD. It is used to trigger the host to issue the first DSM instruction. The first computation unit is the computation result of the forward propagation phase, which is generated by the GPU performing forward computation based on the first data unit. It includes the hierarchical model parameters of that layer and the corresponding layer activation value. It is both the output of the current phase and the input data for the next backpropagation phase.

[0065] In some embodiments of this disclosure, before the SSD controller loads the pre-computation data unit corresponding to the layer identifier to the GPU, the method further includes: sending a second DSM instruction to the SSD controller via an interaction channel. The second DSM instruction is used to instruct the SSD controller to directly load the corresponding first data unit from the SLC to the GPU based on the layer identifier of the layer to be processed, or to first migrate the first data unit from the QLC to the SLC and then load it to the GPU, so that the GPU performs forward computation based on the layer model parameters in the direction from input to output to generate layer activation values, and uses the layer model parameters and the corresponding layer activation values ​​as the first computation unit corresponding to the forward propagation stage.

[0066] In some embodiments, the second DSM instruction is a prefetch instruction (such as the DSMPrefetch instruction) issued by the host to the SSD controller to prompt the SSD about the access attributes of the data to be loaded. After receiving the instruction, the SSD controller can load the corresponding first data unit directly from the SLC cache to the GPU based on the hierarchical identifier of the layer to be processed. If the data is not stored in the SLC cache, the first data unit is first quickly migrated from the QLC storage area to the SLC cache and then loaded to the GPU to reduce data loading latency and ensure the continuity of forward computation.

[0067] In some embodiments, during the forward propagation phase, the GPU performs computation layer by layer in a forward order from the input layer (layer1) to the output layer (layern), generating the corresponding layer activation values, i.e., inter-block activation values. During computation, intra-block activation values ​​are discarded as needed, retaining only the necessary information for recomputation during the backpropagation phase. Inter-block activation values ​​are temporarily stored in the GPU memory. The GPU continuously generates and accumulates activation values ​​for each layer while loading the model parameters for the corresponding network layer. Memory usage gradually increases as computation progresses. When loading the parameters for the current layer, the host sends a DSM Prefetch command to the SSD controller via an interactive channel, marking the layer data as immediately accessible data. The SSD controller prioritizes loading the corresponding data from the SLC cache. If the data is not located in the SLC cache, it first migrates the data from the QLC storage area to the SLC cache before transmitting it to the GPU to reduce data loading latency.

[0068] The GPU uses the loaded parameters to complete the forward computation of the current layer, generates the layer activation value, and temporarily stores it in the GPU memory. When the GPU detects that the memory usage rate has reached a preset threshold, it sends the first information to the host. Based on this, the host sends the first DSM instruction to the SSD controller to set the access priority for each layer's computing unit: such as... Figure 3 As shown, "layer" represents a network layer. During the forward propagation phase, the computation order proceeds from the input to the output, i.e., layer 1-layer 2-...-layern-1-layer n are executed sequentially. During the first data unit loading process, the host sends a second DSM instruction to the SSD controller via the interaction channel, preloading the weights and data units corresponding to the high-priority first network layer (layer 1) into the SLC region. The remaining low-priority network layer data is stored in the QLC region, allowing for rapid migration from QLC to SLC before being transferred to the GPU before reaching the corresponding network layer for computation. During the first computation unit unloading process, the first computation unit closest to the output layer is unloaded to the SLC, such as layer n, and the first computation unit closest to the input layer is unloaded to the QLC, such as layer 1.

[0069] The first computational unit corresponding to the network layer at the model output end (near layer n) is marked as high priority and scheduled to SLC cache storage to ensure the high-speed loading requirements of the subsequent backpropagation stage; the first computational unit corresponding to the network layer at the model input end (near layer 1) is marked as low priority and stored in QLC region, and the data of the remaining intermediate layers are stored as needed to achieve reasonable allocation of GPU memory usage and storage resources.

[0070] In some embodiments of this disclosure, multiple processing stages further include a backpropagation stage in which multiple network layers perform backward computation layer by layer. The corresponding data unit is a second data unit including hierarchical model parameters and hierarchical activation values. Before sending the first DSM instruction to the SSD controller based on the interaction channel, the process includes: receiving second information sent by the GPU. The second information is sent by the GPU before unloading the second computing unit to trigger the host to send the corresponding first DSM instruction. The corresponding first DSM instruction is used to determine the second computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as high priority, and to determine the second computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as low priority. The second computing unit includes gradient values ​​generated by the GPU performing backward computation based on the hierarchical model parameters and hierarchical activation values ​​in the direction from output to input.

[0071] In some embodiments, the corresponding first DSM instruction here refers to a first DSM instruction adapted to the backpropagation phase for prioritizing the second computational units generated in that phase.

[0072] In some embodiments, the second data unit is the input data that the GPU needs to load before performing computation during the backpropagation phase, specifically the layer model parameters and layer activation values ​​corresponding to each network layer generated during the forward propagation phase; the second information is the state trigger information sent by the GPU to the host, sent before the GPU prepares to unload the second computation unit to the SSD, used to trigger the host to issue the first DSM instruction to reconfigure the data access priority; the second computation unit is the computation result of the backpropagation phase, generated by the GPU performing reverse computation based on the second data unit, specifically the gradient values ​​corresponding to each network layer, which is the core input data for the subsequent optimizer update phase.

[0073] In some embodiments, such as Figure 4 As shown, in the backpropagation phase, the computation order is reversed compared to the forward direction, following a backtracking execution from the output to the input, i.e., layer n-layer n-1-...-layer 2-layer1 are executed sequentially. The GPU first reads the previously stored high-priority inter-layer activation values ​​from the output (layer n) in the SLC of the SSD for backpropagation gradient calculation, until it reads the inter-layer activation values ​​from the input (layer 1) in the QLC. During backpropagation, the GPU recalculates the intra-block activation values ​​and combines them with the historical activation values ​​generated in the forward propagation phase and network parameters to perform backpropagation calculation, ultimately generating the corresponding gradient. Frequent data loading is required, making it highly sensitive to data loading speed and latency. Access latency directly restricts overall training efficiency. When loading the corresponding layer activation values ​​and model parameters, it combines... Figure 4In the layout shown, when the corresponding second computing unit is unloaded, the second computing unit closer to the input end (layer 1) is unloaded to the SLC, and the second computing unit closer to the output end (layer n) is unloaded to the QLC.

[0074] The GPU performs reverse computation based on the loaded activation values ​​and parameters to generate corresponding gradient values. Before unloading the second computation unit, the GPU sends a second message to the host. The host sends a first DSM instruction to the SSD controller through the interaction channel to reconfigure the access priority: marking the second computation unit corresponding to the network layer near the model input (near layer 1) as high priority and marking the second computation unit corresponding to the network layer near the model output (near layer n) as low priority. The SSD controller then schedules according to this priority, writing high-priority gradient data to the SLC cache to match the high-frequency demand for reverse access during backpropagation, and writing low-priority gradient data to the QLC storage area, thereby significantly reducing the loading latency of high-frequency data and improving training throughput efficiency.

[0075] In some embodiments of this disclosure, multiple processing stages further include an optimizer update stage that updates the hierarchical model parameters of each network layer. Before sending the first DSM instruction to the SSD controller via the interaction channel, the process includes: receiving third information sent by the GPU. The third information is sent by the GPU before the overall unloading of the third computing unit obtained based on the third data unit, thereby triggering the host to send the corresponding first DSM instruction. The corresponding first DSM instruction is used to determine the third computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as high priority, and to determine the third computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as low priority. The GPU reads the third data unit from the SLC and performs hierarchical model parameters and optimizer state updates based on the third data unit in the direction from input to output, so as to use the updated hierarchical model parameters and optimizer state as the third computing unit.

[0076] In some embodiments, the first DSM instruction here refers to the first DSM instruction used to prioritize the third computational units generated in the adapt optimizer update phase.

[0077] In some embodiments, the third data unit is the input data that the GPU needs to load before performing computation during the optimizer update phase. It consists of the hierarchical model parameters, gradient values, and optimizer state corresponding to each network layer. The third information is the state trigger information sent by the GPU to the host. It is sent before the GPU is ready to unload the third computation unit to the SSD. It is used to trigger the host to issue the first DSM instruction to configure the data access priority for the next iteration. The third computation unit is the computation result of the optimizer update phase. It is generated by the GPU based on the third data unit to perform parameter updates. Specifically, it can be the updated hierarchical model parameters and optimizer state. It is the input data for the forward propagation phase of the next iteration.

[0078] In some embodiments, the GPU executes optimizer algorithms such as Adam based on the loaded third data unit to update the hierarchical model parameters and refresh the optimizer state. The updated hierarchical model parameters and optimizer state are obtained as the third computing unit. Before the third computing unit is unloaded to the SSD layer by layer, the GPU sends third information to the host to trigger the first DSM instruction. The instruction marks the access priority of the third computing unit in the next round of forward propagation. The third computing unit corresponding to the front layer of the model is marked as high-frequency access and written to SLC, and the third computing unit corresponding to the back layer of the model is marked as low-frequency access and written to QLC, thereby forming a storage optimization closed loop in the training cycle.

[0079] In some embodiments of this disclosure, before the SSD controller loads the pre-computation data unit corresponding to the hierarchical identifier to the GPU, the method further includes: sending a third DSM instruction to the SSD controller via an interaction channel. The third DSM instruction is used to instruct the SSD controller to migrate the hierarchical model parameters and optimizer state to the SLC where the gradient value with the same hierarchical identifier is located. The hierarchical model parameters, optimizer state, and gradient value form the third data unit.

[0080] In some embodiments, the third DSM instruction is a data migration instruction issued by the host to the SSD controller, such as the DSM Write Pacing instruction, which instructs the SSD controller to migrate the hierarchical model parameters and optimizer state of the corresponding network layer to the SLC cache where the gradient values ​​with the same hierarchical identifier are located. After the migration is completed, the hierarchical model parameters, optimizer state and gradient values ​​are integrated into the third data unit, which ensures that the GPU can read the complete data from the SLC cache at high speed at one time during the optimizer update stage, ensuring the efficient execution of parameter updates, and preventing the waste of SLC cache resources.

[0081] In some embodiments, such as Figure 5As shown, during the optimizer update phase, the GPU performs parameter updates in a forward order from the input layer (layer 1) to the output layer (layer n). This requires sequential access to the model parameters and corresponding gradients of each layer. The data access order is completely reversed compared to the backpropagation phase. The host computer issues the third DSM instruction and DSMWrite Pacing instruction to the SSD controller based on the training progress, combined with... Figure 5 The storage layout shown pre-migrates the hierarchical model parameters and optimizer states near the input end from the QLC storage area to the SLC cache, placing them in the same storage area as the gradient data of the same level. Unprocessed data near the output end remains in its original partition, thus realizing on-demand allocation of SLC cache resources.

[0082] The GPU reads layer gradients, parameters, and optimizer states from the SLC cache, and updates the parameters and optimizer states in the order from input to output to obtain the updated third computation unit. Before unloading the third computation unit, the GPU sends third information to the host, and the host sends the first DSM instruction to the SSD controller to set the access priority for the next iteration: marking the third computation unit closer to the model output as low priority and the third computation unit closer to the input as high priority. The SSD controller writes the corresponding data to the QLC storage area and the SLC cache respectively, completing the storage scheduling closed loop of the training cycle. By combining the access characteristics of this stage, the data storage strategy is dynamically adjusted to uniformly migrate the gradients, parameters, and optimizer states of the corresponding network layers to the SLC cache, ensuring efficient execution of parameter updates. At the same time, differentiated storage balances SLC resource utilization and flash memory lifespan.

[0083] In this embodiment, considering the differentiated characteristics of memory usage and data access order in the three stages of big data model training—forward propagation, backward propagation, and optimizer update—training data is divided at the network layer level. Data storage is dynamically scheduled at the network layer level by using data units, trigger information, and DSM instructions corresponding to the needs of each stage. Differentiated priority marking and fine-grained storage control prevent storage chaos caused by tensor splitting, adapt to the hierarchical computing logic of big data models, and achieve precise matching between data access requirements and the performance of SLC / QLC storage media. This expands the GPU memory capacity while balancing training efficiency and SSD lifespan.

[0084] In some embodiments of this disclosure, the big data model to be trained is hierarchically divided into multiple network layer identifiers at the network layer level, including: reading the network structure configuration information of the big data model to be trained, and determining multiple network layers and corresponding hierarchical identifiers based on the reading results; and determining at least one of the model parameters, activation values, gradients and optimizer states corresponding to each network layer as data units based on the processing stage.

[0085] In this embodiment, the network structure configuration file of the big data model to be trained is read, all network layer sequences contained in the model are parsed, and a unique and fixed layer identifier is assigned to each layer to distinguish and index the data of different network layers. According to the current training processing stage, one or more of the model parameters, activation values, gradients or optimizer states corresponding to each network layer are encapsulated into data units corresponding to that stage. The model data is organized in a regularized manner with the network layer as the smallest granularity, which is highly matched with the execution flow of the hierarchical computation of the big data model. This prevents irregular splitting of tensors, reduces the addressing overhead and management complexity during data scheduling, and provides a unified data unit basis for subsequent layer-by-layer execution of storage priority marking, loading and unloading, thereby improving the orderliness and targeting of the overall storage scheduling.

[0086] In some embodiments of this disclosure, establishing a DSM command interaction channel with the SSD controller includes: initializing the communication interface with the SSD controller to configure the transmission protocol, data format, and interaction timing of DSM commands; and completing handshake authentication with the SSD controller based on the transmission protocol, data format, and interaction timing to establish the DSM command interaction channel.

[0087] In this embodiment, the physical and protocol communication interface between the host and the SSD controller is initialized. Parameters such as the transmission protocol type, data encapsulation format, and command sending and response timing for DSM commands are uniformly configured. Following the configured protocol, format, and timing, a handshake authentication is performed with the SSD controller to confirm that the command parsing rules and communication parameters are consistent between the two parties, thereby establishing a dedicated interactive channel for DSM command issuance and feedback. This ensures the stability, real-time performance, and parsing consistency of DSM command transmission, preventing command packet loss, format errors, or timing discrepancies. It enables the host to achieve low-latency, fine-grained storage scheduling control over the SSD, providing reliable communication support for subsequent data prefetching, migration, priority marking, and other operations.

[0088] Figure 6 A flowchart of another model training method based on extended video memory is shown in an embodiment of this disclosure.

[0089] like Figure 6 As shown, the SSD controller executes a model training method based on extended GPU memory, including the following steps: Step S602: Establish a DSM command interaction channel with the host. The command interaction channel is used to receive DSM commands of the data access attribute prompt type.

[0090] Step S604: Load the pre-computation data units into the GPU so that the GPU can calculate the computation units based on the data units. The data units correspond to the layer identifiers of the network layers to be trained in the big data model.

[0091] Step S606: Receive the first DSM instruction sent by the host based on the interaction channel.

[0092] Step S608: Based on the first DSM instruction, the priority of the computing unit is marked, and the computing unit with high priority is unloaded to SLC, and the computing unit with low priority is unloaded to QLC to complete a processing stage. The marked computing unit is configured as the data unit to be loaded in the next processing stage. The GPU performs multiple rounds of iterative training, and each round of iterative training includes multiple processing stages.

[0093] In some embodiments of this disclosure, multiple processing stages include a forward propagation stage in which multiple network layers perform forward computation sequentially. The corresponding data unit is a first data unit including hierarchical model parameters. Loading the data unit before computation onto the GPU includes: receiving a second DSM instruction sent by the host based on the interaction channel; based on the second DSM instruction and the hierarchical identifier of the layer to be processed, directly loading the corresponding first data unit from the SLC to the GPU, or first migrating the first data unit from the QLC to the SLC and then loading it onto the GPU, so that the GPU performs forward computation based on the hierarchical model parameters in the direction from input to output to generate hierarchical activation values, and uses the hierarchical model parameters and the corresponding hierarchical activation values ​​as the corresponding first computation unit.

[0094] In some embodiments, prioritizing computation units based on a first DSM instruction and offloading computation units with high priority to the SLC and offloading computation units with low priority to the QLC includes: determining the first computation unit corresponding to the hierarchical identifier between the output end of the big data model and the specified location as high priority based on the first DSM instruction, determining the first computation unit corresponding to the hierarchical identifier between the input end of the big data model and the specified location as low priority, offloading the first computation unit with high priority to the SLC, and offloading the first computation unit with low priority to the QLC.

[0095] In some embodiments of this disclosure, multiple processing stages also include a backpropagation stage in which multiple network layers perform backward computation layer by layer. The corresponding data unit is a second data unit including hierarchical model parameters and hierarchical activation values. Loading the data unit before computation onto the GPU includes: directly transmitting the corresponding second data unit from the SLC to the GPU based on the hierarchical identifier of the layer to be processed, or migrating the second data unit from the QLC to the SLC first, and then transmitting it to the GPU, so that the GPU performs backward computation based on the hierarchical model parameters and hierarchical activation values ​​in the direction from output to input to generate gradient values, and uses the gradient values ​​as the corresponding second computation unit.

[0096] In some embodiments, prioritizing computation units based on a first DSM instruction and offloading high-priority computation units to the SLC and low-priority computation units to the QLC includes: determining a second computation unit corresponding to a hierarchical identifier between the input of the big data model and a specified position as high-priority based on the first DSM instruction, and determining a second computation unit corresponding to a hierarchical identifier between the output of the big data model and a specified position as low-priority based on the first DSM instruction. The second computation unit includes gradient values ​​generated by the GPU performing reverse calculations based on hierarchical model parameters and hierarchical activation values ​​in the direction from output to input. The high-priority second computation units are offloaded to the SLC, and the low-priority second computation units are offloaded to the QLC.

[0097] In some embodiments of this disclosure, multiple processing stages also include an optimizer update stage that updates the hierarchical model parameters of each network layer, and loads the pre-computation data unit onto the GPU, including: receiving a third DSM instruction sent by the host based on an interactive channel; migrating the hierarchical model parameters and optimizer state to the SLC where the gradient value with the same hierarchical identifier is located based on the third DSM instruction, wherein the hierarchical model parameters, optimizer state and gradient value form a third data unit.

[0098] In some embodiments, the priority of computing units is marked based on the first DSM instruction, and computing units with high priority are offloaded to the SLC and computing units with low priority are offloaded to the QLC. This includes: determining the third computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as high priority based on the first DSM instruction, determining the third computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as low priority, offloading the third computing unit with high priority to the SLC, and offloading the third computing unit with low priority to the QLC.

[0099] In some embodiments, the SSD controller establishes a dedicated DSM instruction interaction channel with the host to receive various data access attribute prompts. During multiple iterations of forward propagation, backward propagation, and optimizer updates by the GPU, the data units at each stage are loaded hierarchically according to the hierarchical identifier based on the second DS instruction and the third DSM instruction issued by the host. Data is preferentially read directly from the SLC cache, and if it is not in the SLC, it is migrated from the QLC to the SLC before being supplied to the GPU. Before the GPU completes the computation and unloads the computation unit, the first DSM instruction issued by the host is received. The access priority of the computation unit is marked according to the instruction. High-priority units are written to the SLC cache, and low-priority units are written to the QLC storage area. The marked data units are used as the data to be loaded in the next stage, realizing hierarchical storage scheduling throughout the entire training process.

[0100] In some embodiments of this disclosure, establishing a DSM command interaction channel with a host includes: responding to an initialization configuration operation of the communication interface initiated by the host, obtaining the transmission protocol, data format, and interaction timing of the DSM commands; and completing a handshake authentication with the host based on the transmission protocol, data format, and interaction timing to establish the DSM command interaction channel.

[0101] In some embodiments, the SSD controller responds to the initialization configuration request initiated by the host, obtains and synchronizes the DSM command transmission protocol, data format and interaction timing configured by the host; performs a two-way handshake authentication with the host according to the agreed protocol, format and timing to confirm that the communication parameters are consistent with the command parsing rules; after successful authentication, it establishes a stable interaction channel with the host dedicated to DSM command sending and receiving, providing a reliable command transmission link for subsequent storage scheduling.

[0102] like Figure 7 As shown, the host executes a model inference method based on extended video memory, including: Step S702: The big data model is divided into layers at the network layer level to obtain the model layer. The model layer includes layer weights, and the frequency of use of the layer weights increases from the input end to the output end of the big data model.

[0103] Step S704: Establish a DSM command interaction channel with the SSD controller. The command interaction channel is used to send DSM commands of the data access attribute prompt type to the SSD.

[0104] In step S706, in response to the inference startup operation of the big data model, a second DSM instruction is sent to the SSD controller based on the instruction interaction channel. The second DSM instruction is used to load the first set of weights into the SLC and the second set of weights into the QLC. The first set of weights includes hierarchical weights that are greater than the weight threshold, and the second set of weights includes hierarchical weights that are less than or equal to the weight threshold.

[0105] In some embodiments, the first set of weights and the second set of weights are two sets of weights divided based on the usage frequency of hierarchical weights and a preset weight threshold. The division logic is adapted to the characteristic in step S702 that the usage frequency of hierarchical weights increases from the input end to the output end. That is, the first set of weights are hierarchical weights greater than the weight threshold. These weights correspond to model layers with higher usage frequency and are hotspot weights that are frequently accessed during inference. They are loaded into the SLC high-speed storage area to reduce weight loading latency and match the low latency requirement of inference. The second set of weights are hierarchical weights less than or equal to the weight threshold. These weights correspond to model layers with lower usage frequency and are accessed relatively less frequently. They are loaded into the QLC ordinary storage area. This can ensure normal execution of inference while reasonably allocating storage resources between SLC and QLC, preventing waste of high-speed storage resources, and achieving refined utilization of storage resources.

[0106] In step S708, in response to the obtained inference request, the GPU is triggered to perform inference of the big data model based on hierarchical weights to generate KV cache, and sends DSM instructions corresponding to the type of inference request to the SSD controller based on the instruction interaction channel, so that the SSD writes the KV cache to SLC and / or QLC based on the corresponding DSM instructions.

[0107] In some embodiments of this disclosure, the inference process of the big data model corresponds to the forward propagation process in the model training. As the size of the model weights continues to increase (such as the weights of the Transformer layer, Attention layer, and FFN layer), a single GPU memory can no longer fully store all the weights. Moreover, the KV cache (Key-Value Cache) generated during the inference process will increase dramatically with the input length and the number of concurrent operations, becoming the second largest user of GPU memory. Since inference only involves forward computation and has no backpropagation or optimizer update process, its data access has distinct characteristics. The weights are read-only, sequential or random access, updated infrequently and are highly latency-sensitive. The KV cache is read and written frequently, has local hotspots, a short lifespan, and is extremely sensitive to latency jitter. Therefore, the overall inference process requires low latency, high throughput, and stable response.

[0108] In this embodiment, the inference process of the big data model corresponds only to the forward propagation during training. The GPU reads the layer weights in the order of network layers, processes the input, and continuously generates and updates the KV cache to complete the layer-by-layer inference calculation. Before starting the inference operation, the model is divided according to the granularity of the network layers and hot layer weights are identified. The DSM instruction is used to load the frequently accessed hot weights into the SLC high-speed area and store the low-frequency weights into the QLC ordinary area. During inference, the KV cache is dynamically scheduled to the corresponding storage area according to the request type. This matches the read-only and low-latency sensitive access characteristics of the inference weights with the high-frequency read and write and latency jitter sensitive access characteristics of the KV cache, thereby effectively reducing the overall inference latency and eliminating inference stuttering. While improving the system's concurrent carrying capacity and context inference length, it also achieves efficient utilization of GPU memory and SSD resources.

[0109] In some embodiments of this disclosure, sending a DSM instruction corresponding to the type of inference request to the SSD controller via the instruction interaction channel includes: identifying the inference request as a high-priority request and sending a corresponding first DSM instruction to the SSD controller, the first DSM instruction being used to configure a high-priority storage policy for the KV cache to be written to the SLC; identifying the inference request as a low-priority request and sending a corresponding first DSM instruction to the SSD controller, the first DSM instruction being used to configure a low-priority storage policy for the KV cache to be written to the QLC.

[0110] In some embodiments, the identification of high-priority and low-priority requests can be based on the service requirements, response time limits, and business priorities of the inference request. Specifically, high-priority requests can be understood as requests with strict requirements on inference response latency and requiring fast return of results. They can be identified through preset response time limit thresholds. For example, inference requests with response time limits less than a preset threshold (such as 100ms) or inference requests marked as core business (such as real-time interaction or key decision-making) are all identified as high-priority requests. Low-priority requests can be understood as requests without strict restrictions on inference response latency and allowing for delayed return of results. Specifically, they can be identified as inference requests with response time limits greater than or equal to a preset threshold or non-core business (such as batch queries or background statistics). These requests are identified as low-priority requests. This identification method can achieve differentiated storage scheduling of KV cache to match the latency requirements of different requests.

[0111] In some embodiments of this disclosure, sending a DSM instruction corresponding to the type of inference request to the SSD controller via the instruction interaction channel includes: identifying an inference request as an active request and sending a corresponding third DSM instruction to the SSD controller, wherein the third DSM instruction is used to write the generated KV cache to the SLC; identifying an inference request as an idle request and sending a corresponding third DSM instruction to the SSD controller, wherein the third DSM instruction is used to downgrade the generated historical KV cache to the QLC.

[0112] In some embodiments, the identification of active and idle requests is based on the execution status, interaction frequency, and context continuity of the inference request. Specifically, an active request can be understood as an inference request that is currently performing inference computation, needs to continuously generate or read KV cache, and has continuous context interaction. This can be identified by detecting the interaction frequency of the request (such as multiple association requests initiated within a preset time) and the inference process status (running and not triggering a termination instruction). An idle request can be understood as an inference request that has been paused, has no continuous context interaction, has not initiated a new association request within a preset time, or has completed part of the inference and does not need to access the historical KV cache for the time being. This identification method can dynamically adjust the storage location of the KV cache to prevent SLC resources from being occupied by the cache of idle requests and improve resource utilization.

[0113] In some embodiments of this disclosure, identifying an inference request as an idle request further includes: sending a fourth DSM instruction to the SSD controller via an instruction interaction channel, the fourth DSM instruction being used to instruct the SSD to release the degraded historical KV cache.

[0114] In some embodiments, the fourth DSM instruction is a cache release instruction issued by the host to the SSD controller, which is used to instruct the SSD controller to clean up and release the generated historical KV cache corresponding to the idle request. Specifically, it deletes the historical KV cache data associated with the idle request in the SSD (including SLC and QLC) and releases the occupied storage resources. This instruction can be executed in conjunction with the identification result of the idle request, or it can be triggered after the inference is completed. This prevents invalid historical KV cache from occupying storage resources for a long time, further improves the utilization rate of SSD storage resources, provides sufficient storage space for the KV cache of other active inference requests, and ensures the smoothness of the inference process.

[0115] In some embodiments of this disclosure, the method further includes: in response to an inference termination prompt from the GPU, sending a fourth DSM instruction to the SSD controller, the fourth DSM instruction causing the SSD to release the generated KV cache.

[0116] like Figure 8 As shown, the SSD controller executes a model inference method based on extended video memory, including: Step S802: Establish a DSM command interaction channel with the host. The command interaction channel is used to receive DSM commands of the data access attribute prompt type.

[0117] Step S804: Receive the second DSM instruction sent by the host based on the instruction interaction channel.

[0118] Step S806: Based on the second DSM instruction, load the first set of weights into the SLC and load the second set of weights into the QLC. The first set of weights includes hierarchical weights greater than the weight threshold, and the second set of weights includes hierarchical weights less than or equal to the weight threshold. The hierarchical weights are the weights of the model layers obtained by hierarchically dividing the big data model at the network layer granularity.

[0119] Step S808: Receive the DSM instruction sent by the host based on the instruction interaction channel, corresponding to the type of inference request. The GPU executes the inference operation of the big data model to generate a KV cache.

[0120] Step S810: Write the generated KV cache to SLC and / or QLC based on the corresponding DSM instructions.

[0121] In this embodiment, by establishing a DSM command interaction channel with the host, various data access attribute prompts can be accurately received. Based on the second DSM command, the first set of weights for high-frequency access is loaded into the SLC, and the second set of weights for low-frequency access is loaded into the QLC, matching the usage frequency characteristics of the hierarchical weights and reducing the loading latency of hot spot weights. At the same time, according to the received DSM command corresponding to the inference request type, the KV cache is dynamically written to the corresponding storage area, realizing differentiated storage and scheduling of the KV cache. This not only ensures the low latency requirements of high-priority, active inference requests, but also prevents the waste of high-speed SLC resources, improving the overall storage efficiency and resource utilization of the SSD. In addition, the subsequent fourth DSM command releases the KV cache, further releasing storage resources and ensuring that the SSD can stably adapt to multiple types of inference requests, supporting low latency, high throughput, and stable response in the inference process.

[0122] In some embodiments of this disclosure, receiving a DSM instruction corresponding to the type of inference request sent by the host based on the instruction interaction channel includes: receiving a first DSM instruction; configuring a high-priority storage policy for the KV cache based on the first DSM instruction to write to the SLC; or configuring a low-priority storage policy for the KV cache to be generated to write to the QLC.

[0123] In some embodiments, when the received first DSM instruction carries a high-priority identifier, such as an identifier generated by the host based on the response time limit of the inference request and the service priority, a high-priority storage policy is configured for the KV cache to be generated, and it is written to the SLC high-speed storage area to adapt to the low-latency requirements of high-priority inference requests; when the received first DSM instruction carries a low-priority identifier, a low-priority storage policy is configured for the KV cache to be generated, and it is written to the QLC ordinary storage area to adapt to the storage requirements of low-priority inference requests, thereby realizing differentiated scheduling of KV cache.

[0124] In some embodiments of this disclosure, receiving a DSM instruction corresponding to the type of inference request sent by the host based on the instruction interaction channel includes: receiving a third DSM instruction; using the third DSM instruction to write the generated KV cache into the SLC; or using it to downgrade the generated historical KV cache into the QLC.

[0125] In some embodiments, when the received third DSM instruction carries an active flag (corresponding to an active inference request identified by the host), the instruction is used to instruct the generated KV cache to be written to the SLC, ensuring the high-frequency access efficiency of the active request KV cache; when the received third DSM instruction carries an idle flag (corresponding to an idle inference request identified by the host), the instruction is used to instruct the generated historical KV cache to be downgraded from the SLC to the QLC, releasing the high-speed resources of the SLC, preventing idle cache from occupying high-speed storage, and improving resource utilization.

[0126] In some embodiments of this disclosure, the method further includes: receiving a fourth DSM instruction sent by the host based on the instruction interaction channel, and releasing the degraded historical KV cache based on the fourth DSM instruction.

[0127] In some embodiments, after the SSD controller receives a third DSM instruction carrying an idle identifier and completes the downgrade write of the generated historical KV cache to QLC, if it subsequently receives a fourth DSM instruction from the host, which carries an identifier for the corresponding idle inference request, the SSD controller will release the historical KV cache corresponding to the idle inference request that has been downgraded to QLC (or still remains in SLC) based on the instruction, thoroughly clean up the storage resources occupied by the idle request, further improve the turnover efficiency of SSD storage resources, and provide sufficient storage space for other inference requests.

[0128] In some embodiments of this disclosure, the method further includes: receiving a fourth DSM instruction; and releasing the KV cache in the SLC based on the fourth DSM instruction.

[0129] In some embodiments, the fourth DSM instruction directly received by the SSD controller carries an inference end identifier (which is generated by the host responding to the inference end prompt from the GPU). Based on this instruction, all KV caches generated by the corresponding inference task (including KV caches stored in SLC and QLC) are released directly, quickly clearing invalid caches after inference ends, releasing storage resources in a timely manner, and ensuring the stability and continuity of SSD storage performance.

[0130] like Figure 9 As shown below, based on the interaction between the host, GPU, and SSD controller, the optimization scheme for big data model inference with the model layer as the smallest unit is described, including: In step S902, the host takes the lead, and the SSD controller cooperates to perform inference initialization configuration.

[0131] The host divides the large model to be inferred into layers, with a single network layer as the smallest inference unit. The weight W of each layer is bound to the KV cache space corresponding to that layer, and a unique layer identifier is assigned to each bound layer unit for subsequent data scheduling, storage of tags and issuance of instructions.

[0132] The host establishes an NVMe DSM command interaction channel with the SSD controller, completes channel initialization and communication verification to ensure that the channel can stably transmit preset commands; at the same time, it clarifies the DSM commands and corresponding functions supported by the channel, including Prefetch (weight preloading), Dataset Range (hot / cold data marking), Write Pacing (KV write scheduling), and Deallocate (KV cache release), to provide command support for subsequent inference stages.

[0133] In step S904, the host sends a command, which is executed by the SSD controller to achieve weight loading and preheating.

[0134] After the inference service starts, the host sends a Prefetch (weight preloading) command to the SSD controller through the NVMe DSM channel, instructing the SSD controller to preload the frequently used network layer weights at the top of the model to the SSD's SLC high-speed storage area to achieve low-latency reading; and to directly store the less frequently used weights at the bottom of the model (such as embedding layer and cold branch weights) into the SSD's QLC ordinary storage area to save SLC storage space.

[0135] The host issues the DSM Dataset Range command to mark the weights of each layer that have been loaded into the SSD as hot or cold data, and clarifies the marking rules. High-frequency layers (such as the top Transformer layer, which is frequently called during inference) are marked as hot data and associated with SLC storage; low-frequency layers (such as the bottom embedded layer and cold branches, which are called infrequently during inference) are marked as cold data and associated with QLC storage, providing a basis for subsequent weight reading and scheduling.

[0136] After the SSD controller completes weight loading and marking, it sends the execution status back to the host. The host confirms that all weights have been stored in the corresponding storage areas as required, and that the DSM channel communication is normal.

[0137] Step S906: Host scheduling, GPU execution of inference, and SSD controller responsible for data reading and writing, to execute the inference process.

[0138] When performing single-request or multi-concurrent inference and generating KV cache data, the host controls the KV write strategy of the SSD controller through the DSM Write Pacing (KV write scheduling) command. The KV cache data generated by the currently active inference request (the request that is performing forward computation) is preferentially written to the SSD's SLC high-speed storage area to ensure the lowest possible KV read latency for active requests.

[0139] For KV cache data generated by inference requests that have finished executing or are in an idle state, the DSM command controls either to write it down to the QLC normal storage area of ​​the SSD (which can be read if reused later), or to directly issue the DSMDeallocate command to release this part of the KV cache space and prevent it from occupying SLC resources.

[0140] When performing multi-turn dialogues or long text inference, the GPU needs to read historical key-value cache data to assist in inference. At this time, it prioritizes reading the hot key-value cache (active requests, recently frequently reused key-values) stored in the SLC to achieve low-latency reading and ensure the stability of the inference core latency.

[0141] The cold key-value cache (idle, low reuse rate) stored in QLC is only read when reuse is needed and no hot key-value pairs are available, and does not affect the latency performance of core inference requests.

[0142] During inference, if GPU memory usage reaches a threshold and insufficient memory is encountered, the host computer issues instructions to the SSD controller on a layer-by-layer basis (based on a unique layer identifier) ​​to unload the layer's weight data and corresponding idle key-value cache data to the SSD's QLC storage area. The unloading process preserves the tensor structure of the layer unit. If the layer needs to be accessed again later, it can be reloaded from QLC to SLC or GPU memory via DSM instructions, ensuring uninterrupted inference.

[0143] In step S908, the host takes the lead in scheduling, and the SSD controller performs storage allocation to achieve multi-concurrent inference adaptation.

[0144] The host prioritizes concurrent inference requests, distinguishing between high-priority requests (such as real-time dialogue and inference requests with low latency requirements) and low-priority requests (such as batch inference and requests with low latency requirements).

[0145] The host uses NVMe DSM commands to lock and allocate storage resources for requests of different priorities. Specifically, the model weights and key-value cache data corresponding to high-priority requests are locked in the SLC high-speed storage area of ​​the SSD to ensure that they always enjoy low-latency read and write during inference and are not affected by other requests. The data (weights and key-value cache) of low-priority batch inference requests are all stored in the QLC ordinary storage area of ​​the SSD to make reasonable use of storage resources and prevent them from occupying SLC core space.

[0146] During inference, the host monitors the execution status of each request and the SSD storage usage in real time, and dynamically adjusts the KV write and read strategies through DSM commands to prevent storage resource conflicts between high-priority requests and low-priority requests, thus ensuring inference stability and latency consistency in multi-concurrency scenarios.

[0147] In step S910, the host sends an instruction to the SSD controller to release resources, thus completing the inference loop.

[0148] Once a certain inference request has completed all forward computations and output the inference result, the host immediately sends a Deallocate (KV release) command to the SSD controller through the NVMe DSM channel.

[0149] After receiving the instruction, the SSD controller immediately releases all KV cache data corresponding to the inference request (whether stored in SLC or QLC), clears the storage resources occupied by the request, prevents the SLC space from being wasted by idle data, and reserves sufficient high-speed SLC storage space for the next round of inference requests (especially high-priority requests).

[0150] Repeat steps S906 to S910 to sequentially complete KV writing, reading, memory adaptation, and resource release for each round of inference request, forming a complete inference loop to ensure the continuous and efficient operation of the inference service.

[0151] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0152] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0153] The following reference Figure 10 To describe a first model training apparatus 1000 based on extended video memory according to this embodiment of the invention. Figure 10 The first model training device 1000 based on extended video memory shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0154] The first model training device 1000 based on extended video memory is represented in the form of a hardware module. The components of the first model training device 1000 based on extended GPU memory may include, but are not limited to: a first partitioning unit 1002, used to partition the big data model to be trained into multiple network layer hierarchical identifiers at the network layer level, and store the hierarchical identifiers in the SSD; a first establishment unit 1004, used to establish a DSM instruction interaction channel with the SSD controller, the instruction interaction channel being used to send DSM instructions of the data access attribute hint type to the SSD; and a first sending unit 1006, used to send a first DSM instruction to the SSD controller based on the instruction interaction channel. The GPU performs multiple rounds of iterative training, each round of iterative training including multiple processing stages. Each processing stage includes: the SSD controller loading the pre-computation data units corresponding to the hierarchical identifiers onto the GPU, and the GPU calculating computational units based on the data units. The first DSM instruction is used to mark the priority of the computational units. The processing stage also includes: the GPU unloading computational units from the SSD, so that the SSD controller unloads high-priority computational units to the SLC and low-priority computational units to the QLC. The marked computational units are configured as data units to be loaded in the next processing stage.

[0155] The following reference Figure 11 To describe a second model training device 1100 based on extended video memory according to this embodiment of the invention. Figure 11 The second model training device 1100 based on extended video memory shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0156] The second model training device 1100 based on extended GPU memory is manifested in the form of a hardware module. Components of the second model training device 1100 based on extended GPU memory may include, but are not limited to: a second establishment unit 1102, used to establish a DSM instruction interaction channel with the host, the instruction interaction channel being used to receive DSM instructions of the data access attribute hint type; a first loading unit 1104, used to load pre-computation data units onto the GPU so that the GPU can calculate computation units based on the data units, the data units corresponding to the layer identifiers of the network layers to be trained in the big data model; a first receiving unit 1106, used to receive the first DSM instruction sent by the host based on the interaction channel; and a marking unit 1108, used to mark the priority of computation units based on the first DSM instruction, and unload computation units with high priority to SLC and computation units with low priority to QLC to complete a processing stage, wherein the marked computation units are configured as data units to be loaded in the next processing stage, the GPU performs multiple rounds of iterative training, each round of iterative training including multiple processing stages.

[0157] The following reference Figure 12To describe a first model inference device 1200 based on extended video memory according to this embodiment of the invention. Figure 12 The first model inference device 1200 based on extended video memory shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0158] The first model inference device 1200 based on extended video memory is manifested in the form of a hardware module. The components of the first model inference device 1200 based on extended video memory may include, but are not limited to: a second partitioning unit 1202, used to partition the big data model hierarchically at the network layer level to obtain model layers, the model layers including hierarchical weights, the frequency of use of the hierarchical weights increasing from the input to the output of the big data model; a third establishment unit 1204, used to establish a DSM command interaction channel with the SSD controller, the command interaction channel being used to issue DSM commands of the data access attribute prompt type to the SSD; and a second sending unit 1206, used to respond to the inference start operation of the big data model based on the command interaction channel. A second DSM instruction is sent to the SSD controller. The second DSM instruction is used to load the first set of weights into the SLC and the second set of weights into the QLC. The first set of weights includes hierarchical weights greater than the weight threshold, and the second set of weights includes hierarchical weights less than or equal to the weight threshold. The third sending unit 1208 is used to respond to the acquired inference request, trigger the GPU to perform inference of the big data model based on the hierarchical weights to generate KV cache, and send a DSM instruction corresponding to the type of inference request to the SSD controller based on the instruction interaction channel, so that the SSD writes the KV cache to the SLC and / or QLC based on the corresponding DSM instruction.

[0159] The following reference Figure 13 To describe a second model inference device 1300 based on extended video memory according to this embodiment of the invention. Figure 13 The second model inference device 1300 based on extended video memory shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0160] The second model inference device 1300 based on extended video memory is manifested as a hardware module. Components of the second model inference device 1300 based on extended video memory may include, but are not limited to: a fourth establishment unit 1302, used to establish a DSM instruction interaction channel with the host, the instruction interaction channel being used to receive DSM instructions of the data access attribute prompt type; a second receiving unit 1304, used to receive the second DSM instruction sent by the host based on the instruction interaction channel; a second loading unit 1306, used to load a first set of weights into the SLC and a second set of weights into the QLC based on the second DSM instruction, the first set of weights including hierarchical weights greater than a weight threshold, the second set of weights including hierarchical weights less than or equal to the weight threshold, the hierarchical weights being the model layer weights obtained by hierarchically dividing the big data model at the network layer granularity; a third receiving unit 1308, used to receive the DSM instruction sent by the host based on the instruction interaction channel corresponding to the type of inference request, the GPU executing the inference operation of the big data model to generate a KV cache; and a writing unit 1310, used to write the generated KV cache into the SLC and / or QLC based on the corresponding DSM instruction.

[0161] An electronic device according to an embodiment of the present disclosure includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the extended memory-based model training method or the extended memory-based model inference method described in the above embodiments by executing the executable instructions.

[0162] A storage device according to an embodiment of the present disclosure includes: a controller; an SLC electrically connected to the controller; and a QLC electrically connected to the controller, wherein the controller is configured to execute the model training method based on extended video memory or the model inference method based on extended video memory as described in the above embodiments.

[0163] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0164] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0165] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0166] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0167] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0168] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0169] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0170] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0171] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0172] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A model training method based on extended video memory, characterized in that, This is applied to a host computer, which is connected to a graphics processing unit (GPU) and a solid-state drive (SSD) serving as extended video memory. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC), comprising: The big data model to be trained is divided into multiple network layer hierarchical identifiers at the network layer level, and the hierarchical identifiers are stored in the SSD. Establish a DSM command interaction channel with the SSD controller, the command interaction channel being used to send DSM commands of the data access attribute hint type to the SSD; The GPU sends a first DSM instruction to the SSD controller via the instruction interaction channel. The GPU performs multiple rounds of iterative training, each round of iterative training including multiple processing stages. Each processing stage includes: the SSD controller loading the GPU with the pre-computation data unit corresponding to the level identifier and the GPU calculating the computation unit based on the data unit. The first DSM instruction is used to mark the priority of the computation unit. The processing stage also includes: the GPU unloading the computation unit from the SSD, so that the SSD controller unloads the computation unit with high priority to the SLC and the computation unit with low priority to the QLC. The marked computation unit is configured as the data unit to be loaded in the next processing stage.

2. The model training method based on extended video memory according to claim 1, characterized in that, The multiple processing stages include a forward propagation stage where the multiple network layers perform forward computation sequentially, and the corresponding data unit is a first data unit including hierarchical model parameters. Before the SSD controller loads the pre-computation data unit corresponding to the hierarchical identifier to the GPU, the process further includes: Based on the instruction interaction channel, the SSD controller sends a second DSM instruction. The second DSM instruction is used to instruct the SSD controller to directly load the corresponding first data unit from the SLC to the GPU based on the hierarchical identifier of the layer to be processed, or to first migrate the first data unit from the QLC to the SLC and then load it to the GPU, so that the GPU performs forward computation based on the hierarchical model parameters in the direction from input to output to generate hierarchical activation values, and uses the hierarchical model parameters and the corresponding hierarchical activation values ​​as the first computation unit corresponding to the forward propagation stage.

3. The model training method based on extended video memory according to claim 2, characterized in that, Before sending the first DSM command to the SSD controller via the command interaction channel, the following steps are included: The host receives first information sent by the GPU, which is sent when the GPU detects that the memory usage of the first computing unit has reached a preset threshold, to trigger the host to send the corresponding first DSM instruction. The corresponding first DSM instruction is used to determine the first computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as high priority, and to determine the first computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as low priority.

4. The model training method based on extended video memory according to claim 1, characterized in that, The multiple processing stages include a backpropagation stage where the multiple network layers perform reverse calculations layer by layer. The corresponding data unit is a second data unit including hierarchical model parameters and hierarchical activation values. Before sending the first DSM instruction to the SSD controller based on the instruction interaction channel, the process includes: The host receives a second message sent by the GPU before unloading the second computing unit, which triggers the host to send a corresponding first DSM instruction. The corresponding first DSM instruction is used to determine the second computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as high priority, and to determine the second computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as low priority. The second computing unit includes the gradient value generated by the GPU performing reverse calculation based on the hierarchical model parameters and the hierarchical activation value in the direction from output to input.

5. The model training method based on extended video memory according to claim 1, characterized in that, The multiple processing stages include an optimizer update stage that updates the hierarchical model parameters for each network layer, and before the SSD controller loads the pre-computation data unit corresponding to the hierarchical identifier onto the GPU, the process further includes: Based on the instruction interaction channel, a third DSM instruction is sent to the SSD controller. The third DSM instruction is used to instruct the SSD controller to migrate the hierarchical model parameters and optimizer state to the SLC where the gradient value with the same hierarchical identifier is located. The hierarchical model parameters, the optimizer state and the gradient value form a third data unit.

6. The model training method based on extended video memory according to claim 5, characterized in that, Before sending the first DSM command to the SSD controller via the command interaction channel, the following steps are included: The host receives third information sent by the GPU before the overall unloading of the third computing unit obtained based on the third data unit, to trigger the host to send the corresponding first DSM instruction. The corresponding first DSM instruction is used to determine the third computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position as high priority, and to determine the third computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position as low priority. The GPU reads the third data unit from the SLC and performs the update of the hierarchical model parameters and the optimizer state based on the third data unit in the direction from input to output, so as to use the updated hierarchical model parameters and the optimizer state as the third computing unit.

7. The model training method based on extended video memory according to claim 1, characterized in that, The big data model to be trained is divided into multiple network layer hierarchies at the network layer level, including: Read the network structure configuration information of the big data model to be trained, and determine multiple network layers and corresponding layer identifiers based on the reading results. In this process, at least one of the model parameters, activation values, gradients and optimizer states corresponding to each layer identifier is determined as the data unit.

8. The model training method based on extended video memory according to claim 1, characterized in that, Establish a DSM command interaction channel with the SSD controller, including: Initialize the communication interface with the SSD controller to configure the transmission protocol, data format and interaction timing of the DSM commands; Based on the transmission protocol, the data format, and the interaction timing, a handshake authentication is completed with the SSD controller to establish the instruction interaction channel.

9. A model training method based on extended video memory, characterized in that, An SSD controller is applied to the solid-state drive (SSD) used as the extended video memory. The SSD controller is connected to both the GPU and the host. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC), comprising: Establish a DSM command interaction channel with the host, the command interaction channel being used to receive DSM commands of the data access attribute prompt type; The GPU is loaded with pre-computation data units so that the GPU can compute computation units based on the data units. The data units correspond to the hierarchical identifiers of the network layers to be trained in the big data model. Receive the first DSM command sent by the host based on the command interaction channel; The computing units are prioritized based on the first DSM instruction, and the computing units with high priority are unloaded to the SLC, while the computing units with low priority are unloaded to the QLC to complete a processing stage. The prioritized computing units are configured as the data units to be loaded in the next processing stage. The GPU performs multiple rounds of iterative training, and each round of iterative training includes multiple processing stages.

10. The model training method based on extended video memory according to claim 9, characterized in that, The multiple processing stages include a forward propagation stage, and the corresponding data unit is a first data unit including hierarchical model parameters. Loading pre-computation data units into the GPU includes: Receive the second DSM command sent by the host based on the command interaction channel; Based on the second DSM instruction and the hierarchical identifier of the layer to be processed, the corresponding first data unit is directly loaded from the SLC to the GPU, or the first data unit is first migrated from the QLC to the SLC and then loaded to the GPU, so that the GPU performs forward calculation based on the hierarchical model parameters in the direction from input to output to generate hierarchical activation values, and uses the hierarchical model parameters and the corresponding hierarchical activation values ​​as the corresponding first calculation unit; Prioritizing the computing unit based on the first DSM instruction includes: Based on the first DSM instruction, the first computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position is determined as high priority, and the first computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position is determined as low priority.

11. The model training method based on extended video memory according to claim 9, characterized in that, The multiple processing stages include a backpropagation stage, and the corresponding data unit is a second data unit including hierarchical model parameters and hierarchical activation values. Loading pre-computation data units into the GPU includes: Based on the hierarchical identifier of the layer to be processed, the corresponding second data unit is directly transmitted from the SLC to the GPU, or the second data unit is first migrated from the QLC to the SLC and then transmitted to the GPU, so that the GPU performs reverse calculation based on the hierarchical model parameters and the hierarchical activation value in the direction from output to input to generate gradient value, and uses the gradient value as the corresponding second calculation unit; Prioritizing the computing unit based on the first DSM instruction includes: Based on the first DSM instruction, the second computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position is determined as high priority, and the second computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position is determined as low priority.

12. The model training method based on extended video memory according to claim 9, characterized in that, The multiple processing stages include an optimizer update stage. Loading pre-computation data units into the GPU includes: Receive the third DSM command sent by the host based on the command interaction channel; Based on the third DSM instruction, the hierarchical model parameters and optimizer state are migrated to the SLC where the gradient value with the same hierarchical identifier is located. The hierarchical model parameters, the optimizer state, and the gradient value form a third data unit, so that the GPU generates a third computing unit based on the third data unit. Prioritizing the computing unit based on the first DSM instruction includes: Based on the first DSM instruction, the third computing unit corresponding to the hierarchical identifier between the input end of the big data model and the specified position is determined as high priority, and the third computing unit corresponding to the hierarchical identifier between the output end of the big data model and the specified position is determined as low priority.

13. The model training method based on extended video memory according to claim 9, characterized in that, Establishing a DSM command interaction channel with the host includes: In response to the initialization configuration operation of the communication interface initiated by the host, the transmission protocol, data format and interaction timing of the DSM command are obtained; Based on the transmission protocol, the data format, and the interaction timing, a handshake authentication is completed with the host to establish the instruction interaction channel.

14. A model inference method based on extended video memory, characterized in that, This is applied to a host computer, which is connected to a graphics processing unit (GPU) and a solid-state drive (SSD) serving as extended video memory. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC), comprising: The big data model is divided into model layers at the network layer level. The model layer includes layer weights, and the frequency of use of the layer weights increases from the input end to the output end of the big data model. Establish a DSM command interaction channel with the SSD controller, the command interaction channel being used to send DSM commands of the data access attribute hint type to the SSD; In response to the inference startup operation of the big data model, a second DSM instruction is sent to the SSD controller based on the instruction interaction channel. The second DSM instruction is used to load a first set of weights into the SLC and a second set of weights into the QLC. The first set of weights includes the hierarchical weights that are greater than the weight threshold, and the second set of weights includes the hierarchical weights that are less than or equal to the weight threshold. In response to the acquired inference request, the GPU is triggered to perform inference of the big data model based on the hierarchical weights to generate a key-value cache, and sends a DSM instruction corresponding to the type of the inference request to the SSD controller based on the instruction interaction channel, so that the SSD writes the key-value cache to the SLC and / or QLC based on the corresponding DSM instruction.

15. The model inference method based on extended video memory according to claim 14, characterized in that, Based on the instruction interaction channel, a DSM instruction corresponding to the type of the inference request is sent to the SSD controller, including: The inference request is identified as a high-priority request, and a corresponding first DSM instruction is sent to the SSD controller. The first DSM instruction is used to configure a high-priority storage policy for the KV cache so as to write to the SLC. The inference request is identified as a low-priority request, and the corresponding first DSM instruction is sent to the SSD controller. The first DSM instruction is used to configure a low-priority storage policy for the KV cache so as to write to the QLC.

16. The model inference method based on extended video memory according to claim 14, characterized in that, Based on the instruction interaction channel, a DSM instruction corresponding to the type of the inference request is sent to the SSD controller, including: The inference request is identified as an active request, and a corresponding third DSM instruction is sent to the SSD controller. The third DSM instruction is used to write the generated KV cache into the SLC. If the inference request is identified as an idle request, the corresponding third DSM instruction is sent to the SSD controller. The third DSM instruction is used to downgrade the generated historical KV cache and write it to the QLC.

17. The model inference method based on extended video memory according to claim 16, characterized in that, Identifying the inference request as an idle request also includes: The fourth DSM instruction is sent to the SSD controller via the instruction interaction channel. The fourth DSM instruction is used to instruct the SSD to release the degraded historical KV cache.

18. The model inference method based on extended video memory according to claim 14, characterized in that, Also includes: In response to the inference end notification from the GPU, a fourth DSM instruction is sent to the SSD controller, which causes the SSD to release the generated KV cache.

19. A model inference method based on extended video memory, characterized in that, An SSD controller is applied to the solid-state drive (SSD) used as the extended video memory. The SSD controller is connected to both the GPU and the host. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC), comprising: Establish a DSM command interaction channel with the host, the command interaction channel being used to receive DSM commands of the data access attribute prompt type; Receive the second DSM command sent by the host based on the command interaction channel; Based on the second DSM instruction, the first set of weights is loaded into the SLC, and the second set of weights is loaded into the QLC. The first set of weights includes hierarchical weights that are greater than the weight threshold, and the second set of weights includes hierarchical weights that are less than or equal to the weight threshold. The hierarchical weights are the weights of the model layers obtained by hierarchically dividing the big data model at the network layer granularity. The GPU receives a DSM instruction sent by the host based on the instruction interaction channel, corresponding to the type of inference request of the big data model, and executes the inference operation of the big data model to generate a KV cache. The KV cache is written to the SLC and / or QLC based on the corresponding DSM instruction.

20. The model inference method based on extended video memory according to claim 19, characterized in that, Receiving a DSM instruction sent by the host via the instruction interaction channel, corresponding to the type of inference request of the big data model, including: Receive the first DSM command; The first DSM instruction is used to configure a high-priority storage policy for the KV cache to be written to the SLC; or to configure a low-priority storage policy for the KV cache to be generated to be written to the QLC.

21. The model inference method based on extended video memory according to claim 19, characterized in that, Receiving a DSM instruction sent by the host via the instruction interaction channel, corresponding to the type of inference request of the big data model, including: Receive third DSM commands; The third DSM instruction is used to write the generated KV cache into the SLC; or to downgrade the generated historical KV cache into the QLC.

22. The model inference method based on extended video memory according to claim 21, characterized in that, Also includes: Receive the fourth DSM command sent by the host based on the command interaction channel; The historical KV cache that has been downgraded is released based on the fourth DSM instruction.

23. The model inference method based on extended video memory according to claim 19, characterized in that, Also includes: Receive the fourth DSM command; The KV cache in the SLC is released based on the fourth DSM instruction.

24. A model training device based on extended video memory, characterized in that, This is applied to a host computer, which is connected to a graphics processing unit (GPU) and a solid-state drive (SSD) serving as extended video memory. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC), comprising: The first partitioning unit is used to partition the big data model to be trained into multiple network layer hierarchical identifiers at the network layer granularity, and store the hierarchical identifiers in the SSD; The first establishment unit is used to establish a DSM command interaction channel with the SSD controller. The command interaction channel is used to send DSM commands of the data access attribute prompt type to the SSD. The first sending unit is used to send a first DSM instruction to the SSD controller based on the instruction interaction channel. The GPU performs multiple rounds of iterative training, each round of iterative training including multiple processing stages. The processing stage includes: the SSD controller loading the data unit corresponding to the level identifier to the GPU before computation, and the GPU calculating the computation unit based on the data unit. The first DSM instruction is used to mark the priority of the computation unit. The processing stage also includes: the GPU unloading the computation unit to the SSD, so that the SSD controller unloads the computation unit with high priority to the SLC and unloads the computation unit with low priority to the QLC. The marked computation unit is configured as the data unit to be loaded in the next processing stage.

25. A model training device based on extended video memory, characterized in that, An SSD controller is applied to the solid-state drive (SSD) used as the extended video memory. The SSD controller is connected to both the GPU and the host. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC), comprising: The second establishment unit is used to establish a DSM command interaction channel with the host, and the command interaction channel is used to receive DSM commands of the data access attribute prompt type. The first loading unit is used to load the pre-computation data unit to the GPU so that the GPU can calculate the computation unit based on the data unit. The data unit corresponds to the hierarchical identifier of the network layer to be trained in the big data model. The first receiving unit is configured to receive the first DSM instruction sent by the host based on the instruction interaction channel; A marking unit is used to mark the priority of the computing unit based on the first DSM instruction, and unload the computing unit with high priority to the SLC and the computing unit with low priority to the QLC to complete a processing stage. The marked computing unit is configured as the data unit to be loaded in the next processing stage. The GPU performs multiple rounds of iterative training, and each round of iterative training includes multiple processing stages.

26. A model inference device based on extended video memory, characterized in that, This is applied to a host computer, which is connected to a graphics processing unit (GPU) and a solid-state drive (SSD) serving as extended video memory. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC), comprising: The second partitioning unit is used to partition the big data model into model layers at the network layer level. The model layers include hierarchical weights, and the frequency of use of the hierarchical weights increases from the input end to the output end of the big data model. The third establishment unit is used to establish a DSM command interaction channel with the SSD controller. The command interaction channel is used to send DSM commands of the data access attribute prompt type to the SSD. The second sending unit is used to respond to the inference start operation of the big data model and send a second DSM instruction to the SSD controller based on the instruction interaction channel. The second DSM instruction is used to load a first set of weights into the SLC and a second set of weights into the QLC. The first set of weights includes the hierarchical weights that are greater than the weight threshold, and the second set of weights includes the hierarchical weights that are less than or equal to the weight threshold. The third sending unit is used to respond to the acquired inference request, trigger the GPU to perform inference of the big data model based on the hierarchical weight to generate KV cache, and send a DSM instruction corresponding to the type of the inference request to the SSD controller based on the instruction interaction channel, so that the SSD writes the KV cache to the SLC and / or QLC based on the corresponding DSM instruction.

27. A model inference device based on extended video memory, characterized in that, An SSD controller is applied to the solid-state drive (SSD) used as the extended video memory. The SSD controller is connected to both the GPU and the host. The SSD includes a high-speed storage area (SLC) and a normal storage area (QLC), comprising: The fourth establishment unit is used to establish a DSM command interaction channel with the host, and the command interaction channel is used to receive DSM commands of the data access attribute prompt type. The second receiving unit is used to receive the second DSM instruction sent by the host based on the instruction interaction channel; The second loading unit is used to load the first set of weights into the SLC and the second set of weights into the QLC based on the second DSM instruction. The first set of weights includes hierarchical weights that are greater than the weight threshold, and the second set of weights includes hierarchical weights that are less than or equal to the weight threshold. The hierarchical weights are the weights of the model layers obtained by hierarchically dividing the big data model at the network layer level. The third receiving unit is used to receive the DSM instruction sent by the host based on the instruction interaction channel, which corresponds to the type of the inference request of the big data model, and the GPU executes the inference operation of the big data model to generate a KV cache; The write unit is used to write the KV cache into the SLC and / or QLC based on the corresponding DSM instruction.

28. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute The executable instructions are executed to perform the model training method based on extended video memory as described in any one of claims 1 to 8, or the model inference method based on extended video memory as described in any one of claims 14 to 18.

29. A storage device, characterized in that, include: Controller; SLC is electrically connected to the controller. The QLC is electrically connected to the controller. The controller is used to execute the model training method based on extended video memory as described in any one of claims 9 to 13, or the model inference method based on extended video memory as described in any one of claims 19 to 23.

30. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method based on extended video memory as described in any one of claims 1 to 8, or the model training method based on extended video memory as described in any one of claims 9 to 13, or the model inference method based on extended video memory as described in any one of claims 14 to 18, or the model inference method based on extended video memory as described in any one of claims 19 to 23.

31. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method based on extended video memory as described in any one of claims 1 to 8, or the model training method based on extended video memory as described in any one of claims 9 to 13, or the model inference method based on extended video memory as described in any one of claims 14 to 18, or the model inference method based on extended video memory as described in any one of claims 19 to 23.

Images