Control method and device of training expansion card, storage medium and training all-in-one machine

Abstract

The application discloses a training and inference expansion card control method and device, a storage medium and a training and inference all-in-one machine, and relates to the technical field of electronic digital data processing. The training and inference expansion card control method comprises the following steps: acquiring training and inference data output by a training and inference unit; determining a video memory occupation value of the training and inference data; and if the video memory occupation value is greater than a video memory threshold value associated with the training and inference unit, storing the training and inference data to a training and inference expansion card, so as to reduce the video memory occupation of the training and inference unit. The technical problem that a current training and inference scheme cannot train a model with relatively large parameters under limited video card resources is solved, and the technical effect of large model training and inference under a small video memory scene is achieved.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, and in particular to a control method, device, storage medium, and integrated training and pushing machine for a training and pushing expansion card. Background Technology

[0002] Traditional large-scale model local deployment and training solutions first upload data to a cloud service provider's server cluster, then train the model using interconnected high-performance computing nodes, and finally deploy the generated model to the inference environment. However, this process is highly dependent on the quantity and interconnection efficiency of GPU resources, and the training cost increases superlinearly with the model size. Therefore, current training and inference solutions cannot train models with large parameters with limited GPU resources. Summary of the Invention

[0003] The main purpose of this application is to provide a control method, device, storage medium, and integrated training and pushing machine for a training and pushing expansion card, aiming to solve the technical problem that current training and pushing solutions cannot train models with large parameters under limited graphics card resources.

[0004] To achieve the above objectives, this application provides a control method for a training and push expansion card, the control method comprising:

[0005] Obtain the training and push data output by the training and push unit;

[0006] Determine the video memory usage value of the training and push data;

[0007] If the video memory usage value is greater than the video memory threshold associated with the training and push unit, the training and push data is stored in the training and push expansion card.

[0008] In one embodiment, before the step of acquiring the training and push data output by the training and push unit, the following steps are included:

[0009] Obtain model parameters;

[0010] Based on the video memory specifications, the model parameters are divided into video memory data and expansion card data;

[0011] The expansion card data is stored in the training expansion card, and the video memory data is sent to the training unit.

[0012] In one embodiment, after the steps of storing the expansion card data in the training expansion card and sending the video memory data to the training unit, the method further includes:

[0013] If a pre-caching instruction is received, the target data is determined from the expansion card data according to the pre-caching instruction;

[0014] The target data is sent to the training and push unit.

[0015] In one embodiment, the training and induction data includes at least one of model gradients, optimizer parameters, and checkpoints, and the step of storing the training and induction data to the training and induction expansion card includes at least one of the following:

[0016] Unload the model gradient to the training and pushing extension card;

[0017] The optimizer state is updated based on the optimizer parameters.

[0018] In one embodiment, after the step of unloading the model gradient to the training and pushing extension card, the following steps are included:

[0019] If a gradient update instruction is received from the training and induction unit, the gradient of the target model corresponding to the gradient update instruction is obtained;

[0020] The gradient of the target model is sent to the training and pushing unit.

[0021] In one embodiment, the training and induction data includes model parameters, and the step of storing the training and induction data to the training and induction expansion card includes:

[0022] If the training and inference data also includes checkpoints, the checkpoints are associated with the model parameters and stored in the training and inference extension card.

[0023] In one embodiment, before the step of determining the video memory usage value of the training data, the method further includes:

[0024] Determine whether the training data is used for calculation in the next frame;

[0025] If it is not used for calculation in the next frame, perform the step of determining the video memory usage value of the training data.

[0026] In one embodiment, the control method for the training expansion card further includes:

[0027] If data to be inferred is received, the remaining video memory of the training and inference expansion card is obtained;

[0028] If the remaining video memory is not lower than the operating threshold, the remaining video memory is used to execute the inference process based on the data to be inferred, so as to output the inference result.

[0029] In addition, to achieve the above objectives, this application also provides a training and push integrated machine, which includes a training and push expansion card and a terminal device. The terminal device includes a memory, a processor and a training and push unit. The training and push expansion card interacts with the training and push unit to exchange training and push data by implementing the control method of the training and push expansion card as described above.

[0030] In addition, to achieve the above objectives, this application also provides a control device for a training and pushing expansion card, the control device for the training and pushing expansion card including: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the control method for the training and pushing expansion card as described above.

[0031] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, and stores a program implementing a control method for a training and pushing expansion card. The program implementing the control method for the training and pushing expansion card is executed by a processor to implement the steps of the control method for the training and pushing expansion card as described above.

[0032] This application provides a control method for a training and induction expansion card. This application obtains the training and induction data output by the training and induction unit, and then determines the video memory usage value of the training and induction data. If the video memory usage value is greater than the video memory threshold associated with the training and induction unit, the training and induction data is stored in the training and induction expansion card to reduce the video memory usage of the training and induction unit. This solves the technical problem that current training and induction schemes cannot train models with large parameters under limited graphics card resources, and achieves the technical effect of training and inferring large models in scenarios with small video memory.

[0033] This application reduces the high memory requirements of large model training and inference by scheduling the training and inference data output by the training and inference unit and dynamically allocating it to the GPU memory of the training and inference unit or the training and inference expansion card. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A flowchart illustrating the control method for the training and expansion card in this application (Example 1).

[0037] Figure 2 A flowchart illustrating the second embodiment of the control method for the training and extension card in this application;

[0038] Figure 3 A flowchart illustrating the fifth embodiment of the control method for the training and extension card in this application;

[0039] Figure 4This is a schematic diagram of the hardware structure involved in the control device embodiment of the training and expansion card of this application.

[0040] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0041] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0042] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0043] Currently, traditional large-scale model local deployment and training solutions first upload data to a cloud service provider's server cluster, then train the model using interconnected high-performance computing nodes, and finally deploy the generated model to the inference environment. However, this process is highly dependent on the quantity and interconnection efficiency of GPU resources, and the training cost increases superlinearly with the model size. Therefore, current training and inference solutions cannot train models with large parameters under limited GPU resources.

[0044] The main solution of this application is to acquire the training and inference data output by the training and inference unit; determine the memory usage value of the training and inference data; and if the memory usage value is greater than the memory threshold associated with the training and inference unit, store the training and inference data in the training and inference expansion card. This application reduces the high memory capacity requirement for large model training and inference by scheduling the training and inference data output by the training and inference unit and dynamically allocating it to the memory of the training and inference unit or the training and inference expansion card.

[0045] It should be noted that the execution subject in this embodiment can be a training and push expansion card, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a control device for a training and push expansion card capable of the above functions, or a training and push all-in-one machine including a training and push expansion card. This embodiment does not specifically limit this. The following uses a training and push expansion card as the execution subject as an example to describe this embodiment and the following embodiments.

[0046] Based on this, Embodiment 1 of this application proposes a control method for a training and expansion card. Please refer to... Figure 1 The control method for the training expansion card includes steps S10 to S30:

[0047] Step S10: Obtain the training and push data output by the training and push unit.

[0048] In this embodiment, the training and inference unit is a hardware device used for large model training and inference, encompassing a graphics processor (GPU) and a dedicated acceleration chip. It may also include the GPU's video memory, which is the core component for performing various computational tasks of the model. For example, the training and inference expansion card interacts with the training and inference unit's video memory, or with the GPU of the training and inference unit, or with the CUDA unit. Training and inference data, during model training, includes input training samples, intermediate activation values, checkpoints, model gradients, model parameters, optimizer parameters, etc.; during inference, it includes the data to be inferred, intermediate results, and inference results, serving as the carrier of information during model training and inference.

[0049] As an optional implementation, a data capture mechanism can be set at the output interface of the training and inference unit. For training and inference using a deep learning framework, callback functions or hook mechanisms provided by the framework can be used to capture intermediate results as training and inference data after a specific computation step is completed. For hardware devices, training and inference data can be obtained through the underlying driver. For example, NVIDIA GPUs can use the CUDA programming interface to read relevant data from video memory after the GPU kernel function has finished executing.

[0050] Step S20: Determine the memory usage value of the training data.

[0051] In this embodiment, the video memory occupancy value refers to the storage space occupied by the training and push data in the video memory of the training and push unit, in bytes, reflecting the physical space occupied by the training and push data in the video memory.

[0052] As an alternative implementation, memory usage can be calculated based on the data type and dimension for different data types. Within a deep learning framework, the element size and total number of elements in tensor-type training data can be obtained using methods provided by the framework, thereby calculating the memory usage.

[0053] Step S30: If the video memory usage value is greater than the video memory threshold associated with the training and push unit, the training and push data is stored in the training and push expansion card.

[0054] In this embodiment, the video memory threshold is a quantity limit set for each training and push data unit. Since video memory stores model parameters and other information besides training and push data, the training and push data typically does not directly exceed the total video memory limit; instead, there is a specific setting for training and push data. Furthermore, the video memory threshold is related to the training and push unit; that is, the video memory threshold is positively correlated with the video memory size of the training and push unit. The training and push expansion card is a hardware device that connects to the training and push unit via a high-speed interface. Its main function is to expand the video memory capacity of the training and push unit. When the video memory usage of the training and push data exceeds the video memory threshold, the training and push expansion card can provide additional storage space to store this training and push data, thereby reducing the pressure on the video memory of the training and push unit.

[0055] As an optional implementation, a video memory threshold can be set based on the performance of the training and push unit, model characteristics, and actual system requirements. When the video memory usage of the training and push data calculated by the main controller exceeds the set video memory threshold, the main controller stores the training and push data in a specific file format in the mount point directory corresponding to the training and push expansion card through the operating system's file system interface or a dedicated hardware driver interface.

[0056] Through the above steps, this application realizes the dynamic scheduling of training and inference data output by the training and inference unit, so that it can be reasonably allocated to the training and inference unit's memory or training and inference expansion card according to the memory usage and memory threshold, thereby effectively reducing the high memory capacity requirement for large model training and inference.

[0057] Based on Embodiment 1, Embodiment 2 of this application proposes a control method for a training and pushing expansion card, which includes the following steps before step S10:

[0058] Step A10: Obtain model parameters.

[0059] In this embodiment, model parameters refer to a set of numerical values ​​that determine the model's functionality and performance in a large model. For example, in a neural network, model parameters include the connection weights between neurons, bias values, etc. These parameters are continuously adjusted during training so that the model can learn patterns and rules in the data, thereby achieving accurate prediction or classification of new data.

[0060] As an alternative implementation, model parameters can be read from storage devices, the internet, or cloud storage. Model parameters are typically stored as files on local hard drives, network storage devices, or cloud storage. These storage devices can be accessed and the model parameter files read via file system interfaces or network interfaces. For example, in common deep learning frameworks, model parameter files may be saved in formats such as .pth (PyTorch) and .ckpt (TensorFlow).

[0061] As an alternative implementation, in a distributed training environment, model parameters are synchronously obtained from multiple nodes. Distributed communication protocols (such as MPI, NCCL, etc.) are used to achieve parameter synchronization between nodes.

[0062] Step A20: Based on the video memory specifications, the model parameters are divided into video memory data and expansion card data.

[0063] In this embodiment, the video memory specification refers to the physical characteristics of the training and push unit's video memory, such as its capacity and bandwidth. The video memory specification determines the amount of data the video memory can store and process, as well as the data transfer speed. Video memory data is a portion of the data allocated from the model parameters. This portion of data is within the capacity of the training and push unit's video memory and can be directly stored in the video memory for rapid computation and processing by the training and push unit. Expansion card data is also allocated from the model parameters. Because its size exceeds the capacity of the training and push unit's video memory, or to optimize video memory usage efficiency, this portion of data is stored in the training and push expansion card.

[0064] As an optional implementation, the total capacity and currently occupied capacity of the training and push unit's video memory are obtained by querying hardware information or relevant interfaces provided by the operating system, and then the available video memory capacity is calculated. Based on the overall scale of the model parameters and the available video memory capacity, the model parameters are segmented. The portion of model parameters within the available video memory capacity is determined as video memory data, while the remaining model parameters exceeding the available video memory capacity are classified as expansion card data.

[0065] For example, if the available video memory is 4GB and the total size of the model parameters is 10GB, then the first 4GB of model parameters are used as video memory data, and the last 6GB are used as expansion card data.

[0066] For example, if the available video memory capacity is 16GB and the total size of the model parameters is 648GB, then the first 16GB of model parameters are used as video memory data, and the last 632GB are used as expansion card data.

[0067] Step A30: Store the expansion card data in the training expansion card and send the video memory data to the training unit.

[0068] In this embodiment, the expansion card data is saved to the storage chip of the training expansion card so that it can be read and used later when needed. The video memory data is transferred from the storage device or other nodes to the video memory of the training unit, enabling the training unit to directly access and process this data.

[0069] As an optional implementation, the data from the training and push expansion card is written to its storage medium using an interface provided by the card (such as a PCIe interface). During the writing process, data formatting and verification operations are performed to ensure the integrity and correctness of the data during storage. High-speed data transfer interfaces (such as the GPU bus) are used to transfer video memory data from storage devices or other nodes to the training and push unit's video memory. During transmission, the data transfer order and rate are rationally planned to fully utilize data transfer bandwidth and improve transmission efficiency.

[0070] By taking the steps described above, the model parameters are properly processed and allocated at the beginning of training, providing a solid data foundation for subsequent training and inference processes. Combined with subsequent training and inference data processing steps, the use of GPU memory during large model training and inference can be effectively optimized, reducing the demand for high-capacity GPU memory.

[0071] Optionally, after step A30, the following steps are included:

[0072] Step A40: If a pre-caching instruction is received, determine the target data in the expansion card data according to the pre-caching instruction.

[0073] In this embodiment, the pre-caching instruction is a system-issued instruction to preload certain data from the training and pushing expansion card into the training and pushing unit's video memory. This instruction is typically generated based on the analysis and prediction of the training and pushing unit's subsequent computational tasks, aiming to reduce data waiting time during the training and pushing process and improve computational efficiency. The target data is the model parameter data selected from the expansion card data according to the pre-caching instruction, which the training and pushing unit will need to use in the subsequent computations.

[0074] As an optional implementation, upon receiving a pre-caching instruction, the instruction is parsed to extract the identification information regarding the required model parameters. Based on this identification information, a search and matching process is performed in the expansion card data already stored in the training and propellant expansion card to determine the target data. For example, if the pre-caching instruction specifies the need for the weight parameters of a certain layer of a neural network, then the weight parameters of that layer are located in the expansion card data as the target data.

[0075] Step A50: Send the target data to the training and push unit.

[0076] In this embodiment, the data transmission interface (such as a PCIe interface) between the training expansion card and the training unit is used to read the determined target data from the storage medium of the training expansion card and send it to the video memory of the training unit. During the transmission process, similar to the method used for transmitting video memory data, the data transmission is optimized, such as by arranging the transmission order and controlling the transmission rate, to improve the efficiency and reliability of data transmission.

[0077] Through the above steps, model parameters are appropriately processed and allocated at the start of training, and subsequently, the required expansion card data is loaded into the training and inference unit's GPU memory according to pre-caching instructions, providing necessary data support for the training and inference unit's computation. Combined with subsequent training and inference data processing steps, the use of GPU memory for large model training and inference can be effectively optimized, reducing the demand for high-capacity GPU memory and improving overall training and inference efficiency.

[0078] Based on any of the above embodiments, Embodiment 3 of this application proposes a control method for a training and push expansion card, wherein the step of storing the training and push data in the training and push expansion card includes at least one of the following:

[0079] Step S31: Unload the model gradient to the training and pushing extension card.

[0080] In this embodiment, the training and induction data includes at least one of the following: model gradient, optimizer parameters, and checkpoints. Model gradient: In large model training, the model gradient is the partial derivative of the objective function with respect to the model parameters. It represents the rate of change of the objective function with respect to the current parameter values, reflecting which direction adjusting the model parameters can reduce the objective function value, and is an important basis for updating model parameters. Optimizer parameters: The optimizer is used to adjust model parameters to minimize the loss function. Optimizer parameters are the parameters used by the optimizer during the update process, such as learning rate and momentum. Different optimizers have different parameter settings, which affect the training speed and effect of the model. Checkpoint: A checkpoint is a snapshot of the model during training, containing the model's state at a specific moment, such as the model's parameter values ​​and the optimizer's state. Saving checkpoints facilitates resuming training after an interruption and can also be used for model evaluation and comparison. Unloading: The process of moving data from the GPU memory of the training and induction unit to the training and induction expansion card. The purpose is to free up GPU memory space in the training and induction unit to cope with insufficient GPU memory. Optimizer state: The internal state of the optimizer during training. For example, when using a stochastic gradient descent (SGD) optimizer, the accumulated momentum is part of the optimizer state. The optimizer state is continuously updated as training progresses to guide the updating of model parameters.

[0081] As an optional implementation, when the training and induction data output by the training and induction unit contains model gradients, and the memory usage of the calculated model gradients exceeds the memory threshold associated with the training and induction unit, the following operations are performed: Data Tagging: A unique identifier is added to the model gradient data, containing information such as data type (model gradient), generation time, and model layer, facilitating subsequent management and retrieval on the training and induction expansion card. Data Transfer: The model gradient data is transferred from the memory of the training and induction unit to the storage medium of the training and induction expansion card via a high-speed interface (such as a PCIe interface). During transmission, a data compression algorithm (such as Huffman coding) is used to compress the model gradient data to reduce the amount of data transmitted and the transmission time. Storage Management: Storage space is allocated for the model gradient data in the training and induction expansion card, and its storage location and related metadata (such as identifier, size, etc.) are recorded. Storage management can be performed using a file system or database to quickly locate and access the model gradient data.

[0082] Step S32: Update the optimizer state based on the optimizer parameters.

[0083] As an optional implementation, when the training and induction data output by the training and induction unit contains optimizer parameters, the following operations are performed: Parameter parsing: The optimizer parameters are parsed to extract the values ​​of each parameter, such as learning rate and momentum. Different optimizers have different parameter formats, requiring parsing according to the specific optimizer type. State update: The internal state of the optimizer is updated based on the parsed optimizer parameters. For example, when using a stochastic gradient descent optimizer with momentum, the accumulated momentum value is updated according to the new momentum parameter. The update process must follow the optimizer's update rules to ensure the correctness of the optimizer state. State saving: The updated optimizer state is saved to the video memory of the training and induction unit or to the training and induction expansion card. If there is sufficient video memory, the optimizer state can be saved to video memory for fast access; if video memory is limited, the optimizer state is saved to the training and induction expansion card, and its storage location is recorded.

[0084] For example, the training and push expansion card is connected to the GPU via a PCIe interface. During training, video memory is limited, requiring proper management of training and push data.

[0085] The model is trained using the training dataset. In each training step, the model's loss function is calculated, and the model gradient is computed using backpropagation. Memory usage is calculated: the memory usage of the model gradients is calculated. Assuming a memory threshold of 4GB associated with the training and induction units, a data offloading operation is triggered when the memory usage of the model gradients exceeds 4GB. An identifier is added to the model gradient data, such as "gradient_20250226_1000_layer3", indicating that this is the gradient of layer 3 generated at 10:00 AM on February 26, 2025. The model gradient data is compressed using Huffman coding, and the compressed data is transferred to the training and induction expansion card via the PCIe interface. A folder named "gradients" is created in the file system of the training and induction expansion card to store the model gradient data, and its storage path and related metadata are recorded. A stochastic gradient descent optimizer with momentum is used. In each training step, the optimizer's learning rate and momentum parameters are updated. The optimizer parameters are parsed, and the optimizer's internal state is updated based on the new parameter values, such as updating the accumulated momentum value. Due to limited video memory, the updated optimizer state is saved to the training and push expansion card, and the storage location of the optimizer state in the training and push unit's video memory is recorded.

[0086] Through the above embodiments, the control method of the training and pushing expansion card is applied in actual model training scenarios to reasonably store and manage training and pushing data, so as to reduce the demand for high-capacity GPU memory for large model training.

[0087] Optionally, after step S31, the following steps are also included:

[0088] Step S311: If a gradient update instruction is received from the training and induction unit, obtain the target model gradient corresponding to the gradient update instruction;

[0089] Step S312: Send the gradient of the target model to the training and pushing unit.

[0090] In this embodiment, the gradient update instruction is a command issued by the training and induction unit to request specific model gradients for gradient update operations. This instruction includes relevant identifiers of the required model gradients, such as generation time and the model layer to which they belong, to accurately locate the target model gradient. The target model gradient is the model gradient data that the training and induction unit needs to use during the gradient update process, retrieved and determined from the training and induction extension card according to the gradient update instruction. Accumulated gradients: In some deep learning training strategies, to simulate larger batch sizes, model gradients are gradually accumulated over multiple training steps, and then used to update model parameters after a certain number of steps.

[0091] As an optional implementation, the training and push extension card listens for instructions from the training and push unit in real time. Upon receiving a gradient update instruction, it parses the instruction and extracts the identifier information of the target model gradient, such as generation time and model layer. Based on the parsed identifier information, it searches for the gradient in the training and push extension card's storage management system. When storing model gradient data, the training and push extension card records relevant metadata (such as identifier and storage location) for each data point, allowing for quick location of the target model gradient's storage location. After locating the target model gradient data, it verifies it to ensure data integrity and correctness. Verification can be performed using checksums, hash values, etc. If data verification fails, appropriate error handling measures are required, such as re-acquiring the data or reporting error information. The target model gradient data is read from the training and push extension card's storage medium. If the data was compressed during storage, such as using Huffman coding, decompression is required after reading the data. The read and decompressed target model gradient data is sent to the training and push unit's video memory via a high-speed interface. During transmission, some optimization strategies can be adopted, such as data block transmission and parallel transmission, to improve transmission efficiency.

[0092] For example, a recurrent neural network (RNN) model is trained using the TensorFlow framework for a text generation task. The training and induction unit is an AMD GPU, and the training and induction expansion card is connected to the GPU via a PCIe 4.0 interface. During training, to reduce GPU memory pressure, model gradients are unloaded onto the training and induction expansion card. A cumulative gradient training strategy is employed, with model parameters updated every four training steps. In each training step, the training and induction unit calculates the model's loss function and computes the model gradient using the backpropagation algorithm. When the GPU memory usage of the calculated model gradients exceeds the GPU memory threshold associated with the training and induction unit, the model gradients are unloaded onto the training and induction expansion card. A unique identifier is added to each model gradient data point, such as "gradient_20250226_1400_layer5", indicating that this is the gradient of layer 5 generated at 14:00 on February 26, 2025. After every four training steps, the training and induction unit needs to update the model parameters, at which point a gradient update command is issued. This command contains the identifier information of all model gradients generated in those four training steps. After receiving the gradient update command, the training and induction extension card parses the command and extracts the identifier information of the four model gradients. Based on the identifier information, it locates the corresponding model gradient data file in the training and induction extension card's file system. Each model gradient data file is stored in a folder named "gradients" and categorized according to the identifier information. The four model gradient data files are read from the training and induction extension card's storage medium and decompressed. The decompressed model gradient data is then transferred in blocks and in parallel to the training and induction unit's video memory via the PCIe 4.0 interface. After receiving all model gradient data, the training and induction unit sends an acknowledgment message to the training and induction extension card, completing the data transmission task required for this gradient update. Gradient accumulation and model update: The training and induction unit accumulates the received four model gradients in the video memory and then uses the accumulated gradients to update the model parameters to continue the next round of training.

[0093] Through the above embodiments, the gradient of the target model is obtained from the training and pushing expansion card according to the gradient update instruction and sent to the training and pushing unit to support the training strategy of accumulating gradients, while effectively reducing the demand for high-capacity GPU memory for training large models.

[0094] Based on any of the above embodiments, Embodiment 4 of this application proposes a control method for a training and push expansion card, wherein the step of storing the training and push data to the training and push expansion card includes:

[0095] Step S33: If the training and induction data also includes checkpoints, store the checkpoints and model parameters together in the training and induction expansion card.

[0096] In this embodiment, the training and inference data includes model parameters.

[0097] As an optional implementation, checkpoints and model parameters are stored in the training extension card in an interrelated manner, so that when needed later, the corresponding model parameters can be quickly located based on the checkpoints, or the corresponding checkpoint information can be found based on the model parameters.

[0098] For example, the training and inference unit monitors the model's computational tasks in real time, analyzing whether each model parameter in the current GPU memory will be used in subsequent computation steps. This can be determined by tracking the model's computational graph and analyzing the training and inference processes. For instance, during the training of some deep neural networks, parameters of certain layers may not be needed for computation at specific training stages, and the training and inference unit can identify these parameters. When it is determined that there are model parameters in the GPU memory that do not need to be used for computation, these model parameters are transferred from the training and inference unit's GPU memory to the training and inference expansion card. A unique identifier is added to the model parameters to be unloaded, containing information such as the model layer to which the parameter belongs and the generation time, facilitating subsequent management and retrieval on the training and inference expansion card. The model parameters are transferred from the training and inference unit's GPU memory to the storage medium of the training and inference expansion card using a high-speed interface. During the transfer, the model parameters are compressed to reduce the amount of data transferred and the transfer time. Storage space is allocated for the model parameters on the training and inference expansion card, and their storage location and related metadata are recorded. Storage management can be performed using a database or file system to quickly locate and access the model parameters. Before storing model parameters and checkpoints, establish the relationships between them. This can be achieved by creating a relational table in the database or using specific naming conventions in the file system. For example, create a relational table in the database to record the identifiers of model parameters and their corresponding checkpoints; in the file system, checkpoint files and model parameter files can be named with the same prefix to indicate their relationship. Update the relevant metadata in the training expansion card's storage management system, including the storage locations and relationships of checkpoints and model parameters. This way, when needed later, the relevant data can be quickly located and retrieved based on the metadata.

[0099] For example, during training, GPU memory is limited, necessitating proper management of model parameters. Simultaneously, checkpoints are periodically saved to facilitate training recovery and model evaluation. The language model is trained using the training dataset. In each training step, the training-introduction unit analyzes whether each model parameter in the current GPU memory needs to participate in the computation in subsequent steps. For instance, during the computation of the multi-head attention mechanism, it is found that some feedforward neural network parameters are not needed in the current step. These unnecessary model parameters are marked with identifiers, such as "param_20250226_1600_layer8", indicating that these are the model parameters for layer 8 generated at 16:00 on February 26, 2025. The model parameters are compressed using the LZ77 compression algorithm, and the compressed data is transferred to the training-introduction expansion card via the PCIe 5.0 interface. A folder named "parameters" is created in the file system of the training-introduction expansion card, the model parameter files are stored in this folder, and their storage path and related metadata are recorded. Save a checkpoint every 100 training steps. At the end of the 100th step, save the current model's checkpoint, including model parameter values, optimizer status, and other information. Add an identifier to the checkpoint, such as "checkpoint_20250226_1600_step100". Create a relational table in the database to record the relationship between "param_20250226_1600_layer8" and "checkpoint_20250226_1600_step100". Create a folder named "checkpoints" in the training and push extension card's file system, store the checkpoint files in this folder, and record their storage path and related metadata. Update the metadata in the training and push extension card's storage management system, including the storage location and relational relationship of checkpoints and model parameters.

[0100] The above examples demonstrate how, in real-world model training scenarios, model parameters that do not require computation can be offloaded to the training and push expansion card, and checkpoints can be associated with and stored with model parameters. This effectively reduces the high memory requirements for large model training and facilitates training recovery and model evaluation.

[0101] Based on any of the above embodiments, Embodiment 5 of this application proposes a control method for a training and push expansion card. Before step S20, the method includes: determining whether the training and push data is used for calculation in the next frame; if it is not used for calculation in the next frame, the method of determining the video memory usage value of the training and push data is executed.

[0102] In this embodiment, the next frame calculation refers to the calculation steps performed sequentially in chronological order during the model's training or inference process. The next frame calculation refers to the calculation step immediately following the current one. Whether the training and inference data is used for the next frame calculation determines whether that data needs to be immediately retained in the training and inference unit's memory to meet the requirements of fast access.

[0103] As an alternative implementation, the computational graph of large models is extremely complex, containing thousands or even tens of thousands of nodes and edges. Advanced graph analysis algorithms are used to deeply analyze the computational graph, clarifying the dependencies and usage positions of each training and inference data point. For example, in a large language model based on the Transformer architecture, the output data of certain intermediate layers may only be used in specific multi-head attention mechanisms or feedforward network modules. By analyzing the computational graph, it can be accurately determined whether this data will be used in the next frame's computation. During the training and inference process of large models, the computational progress and status are tracked in real-time and accurately. Using timestamps and event logs in the distributed system, the generation and usage times of each training and inference data point are recorded. Combined with the current computational step, it can be accurately determined whether the data will be used in the next computation. For example, in the distributed training of large models, the intermediate gradient data generated by some nodes may only be used for local updates in the current iteration step and will not be used in the next frame's computation. When it is determined that the training and inference data will not be used in the next frame's computation, the following operations are performed to determine its memory usage: The data types in large models are diverse and complex, including not only common integers, floating-point numbers, and booleans, but also custom data types. A specialized data type identification algorithm is employed to accurately identify the data type of the training and inference data. For example, for quantized data in large models, the algorithm accurately identifies the quantization bit depth and data format to determine the storage space occupied by each element. Data in large models typically has extremely high dimensionality; for example, the dimensions of tensors may reach tens or even hundreds. An efficient dimensionality calculation algorithm is used to accurately calculate the size of each dimension of the multidimensional data. The memory usage of the training and inference data is obtained by multiplying the number of bytes occupied by the data type by the total number of elements (i.e., the product of the sizes of each dimension). For example, for a 16-bit floating-point tensor with the shape (1024, 1024, 1024, 64), the total number of elements is 1024 × 1024 × 1024 × 64, and each element occupies 2 bytes. Therefore, the memory usage of this tensor is 1024 × 1024 × 1024 × 64 × 2 bytes. When the memory usage of the training and push data exceeds the memory threshold, the following steps are performed to store the training and push data in the training and push expansion card: A unique and detailed identifier is added to the training and push data, containing rich information such as data type, generation time, model layer, and data source node. Simultaneously, an efficient data index structure is constructed to facilitate rapid management and retrieval in the training and push expansion card. For example, a hash table or B-tree data structure is used to store the mapping relationship between data identifiers and storage locations. The training and push data is transferred from the graphics memory of the training and push unit to the storage medium of the training and push expansion card via a high-speed, low-latency interface (such as PCIe 5.0 or higher). Advanced data compression algorithms (such as Zstd compression algorithm) are used to efficiently compress the training and push data, significantly reducing data transmission volume and time. Furthermore, multi-channel parallel transmission technology is utilized to further improve transmission efficiency.Due to the massive amount of training and push data generated by large models, the training and push expansion card may employ a distributed storage architecture. When storing training and push data, a distributed file system (such as Ceph, GlusterFS, etc.) is used for management, distributing the data across multiple storage nodes. Simultaneously, the storage location and related metadata (such as identifier, size, checksum, etc.) of each data block are recorded to ensure data reliability and accessibility.

[0104] For example, the DeepSpeed ​​framework is being used to train a large language model with 1000B parameters. The training and induction unit is a cluster of 64 NVIDIA A100 GPUs, with the training and induction expansion card connected to the GPUs via a high-speed PCIe 5.0 interface, employing a distributed storage architecture. In each training step, the output data of each layer of the model is captured as training and induction data through a customized hook mechanism within the DeepSpeed ​​framework. For instance, data collection code is added to the forward propagation methods of the model's self-attention layer and feedforward network layer to collect the output results of each layer. A graph analysis algorithm is used to perform deep analysis of the computational graph of the large language model, revealing that the output data of some intermediate layers is no longer used after subsequent layer normalization operations, i.e., it is not used for the next frame's computation. The computational status of each GPU node in the cluster is tracked in real time, recording the generation and usage times of each training and induction data point to further confirm that this data will not be used in the next computation. For training and induction data determined not to be used for the next frame's computation, its data type is precisely identified as 16-bit floating-point numbers, and its dimension is calculated. Assuming the output tensor of a certain intermediate layer has a shape of (2048, 4096, 1024), its total number of elements is 2048 × 4096 × 1024, and each element occupies 2 bytes. Therefore, the video memory usage of this tensor is 2048 × 4096 × 1024 × 2 bytes. If the video memory usage exceeds the video memory threshold associated with the training and pushing unit, the training and pushing data is stored in the training and pushing expansion card. A detailed identifier "output_20250226_2000_layer5_node12" is added to this training and pushing data, indicating that this is the output data of the 5th layer, from the 12th node, generated at 20:00 on February 26, 2025. Simultaneously, this identifier is added to the data index structure. The training and pushing data is efficiently compressed using the Zstd compression algorithm, and the compressed data is quickly transmitted to the distributed storage system of the training and pushing expansion card through the multi-channel parallel transmission technology of the PCIe 5.0 interface. In the distributed file system, storage space is allocated for the training data, the data blocks are distributed and stored on multiple storage nodes, and the storage location and related metadata of each data block are recorded.

[0105] Through the above embodiments, it is determined whether the training and push data is used for the calculation of the next frame, and the unused data is stored in the training and push expansion card according to the memory usage value and memory threshold. This effectively reduces the demand for high memory capacity for large model training and improves training efficiency and resource utilization.

[0106] Based on any of the above embodiments, Embodiment Six of this application proposes a control method for a training and pushing expansion card, which further includes:

[0107] Step B10: If the data to be inferred is received, obtain the remaining video memory of the training expansion card.

[0108] In this embodiment, the data to be inferred refers to the data that needs to be input into a large model for inference calculations in a real-time training (training while inference) scenario. For example, in a natural language processing task, the data to be inferred might be a piece of text; in an image recognition task, it might be an image. The remaining video memory of the training-inference expansion card refers to the amount of unused storage space in the current video memory of the training-inference expansion card. Since the training-inference expansion card is used to store some training-inference data to alleviate the video memory pressure on the training-inference unit, its remaining video memory reflects the available resources currently available for executing the inference process. The operating threshold is a minimum available space standard set for the video memory of the training-inference expansion card. When the remaining video memory is not lower than this threshold, it indicates that the training-inference expansion card has sufficient resources to execute the inference process; otherwise, inference may not be possible.

[0109] Step B20: If the remaining video memory is not lower than the running threshold, the remaining video memory is used to execute the inference process based on the data to be inferred, so as to output the inference result.

[0110] In this embodiment, the inference process is the process by which the large model calculates based on the input data to be inferred, using the learned model parameters, to derive the inference result. In real-time training scenarios, the inference process and the training process run in parallel.

[0111] As an optional implementation, the training and inference unit interacts with the training and inference expansion card via a specific communication protocol (such as the PCIe interface communication protocol). The training and inference unit sends a request command to the training and inference expansion card to obtain the available video memory. After receiving the request command, the training and inference expansion card queries the video memory usage through its internal hardware management module. This module can directly access the video memory status register to obtain information on the used video memory space and the total video memory space. Based on the queried used video memory space and total video memory space, the available video memory of the training and inference expansion card is calculated. For example, if the total video memory of the training and inference expansion card is 1024GB and 20GB has been used, then the available video memory is 1004GB. When the available video memory of the training and inference expansion card is not lower than the operating threshold, the data to be inferred is loaded from the input device (such as a network interface, storage device, etc.) into the video memory of the training and inference expansion card. During the loading process, the data may need to be preprocessed, such as normalization, pruning, encoding, etc., to make it conform to the input requirements of a large model. The training and inference expansion card retrieves the relevant parameters of the large model from its storage and loads them into the video memory. Due to the massive number of parameters in the large model, techniques such as parameter partitioning and asynchronous loading may be necessary to improve loading efficiency. The training and inference expansion card invokes its computing cores (such as GPU cores or dedicated inference chips) to perform inference computation based on the loaded data to be inferred and the model parameters. During the computation, parallel computing techniques, such as multi-threaded computation and SIMD instruction sets, are used to accelerate the inference speed. After the inference computation is completed, the obtained inference results are output from the training and inference expansion card's video memory. The results can be returned to the requesting end (such as a client or application) or used for further processing and analysis.

[0112] For example, a real-time training and inference task is being performed on a massive natural language processing (NLP) model with 1000 bytes of parameters. Traditionally, this would require 32 NVIDIA A100 graphics cards to meet the training and inference needs. However, with the addition of a training-introduction expansion card, only four NVIDIA A100 graphics cards are needed as training and introduction units. The expansion card connects to the training and introduction units via a high-speed PCIe 5.0 interface, with an operating threshold set at 50GB. The NLP system receives a large amount of user-input text data as inference data, covering various domain-specific questions and sentence comprehension requirements. The training and introduction units send a request to the expansion card via the PCIe 5.0 interface to obtain available video memory. The expansion card's hardware management module queries the video memory status register, finding a total of 512GB of video memory, with 220GB already used, calculating a remaining memory of 292GB. Since the remaining 292GB of video memory is not less than the operating threshold of 50GB, the expansion card begins the inference process. Multi-threaded parallel technology is used to rapidly load the text data to be inferred from a high-speed network interface into the GPU memory of the training and inference expansion card. Simultaneously, preprocessing operations such as word segmentation, part-of-speech tagging, and syntactic analysis are performed on the data. Parameter block storage and asynchronous loading techniques are employed to quickly load relevant parameters of the large language model from the large storage capacity of the training and inference expansion card into the GPU memory. The training and inference expansion card utilizes its multi-core GPU cores and dedicated inference chip array to perform large-scale inference computations based on the loaded text data and model parameters, employing CUDA parallel programming models and tensor parallel computing technology to accelerate the inference process. After the inference computation is completed, the generated accurate response text and semantic understanding results are output as the inference results from the GPU memory of the training and inference expansion card and returned to the user, meeting the user's consultation and understanding needs.

[0113] Through the above embodiments, in the scenario of large-scale models with real-time training (training while inference), the decision to execute the inference process is based on the remaining memory of the training and inference expansion card. This fully leverages the large capacity of the training and inference expansion card to achieve efficient inference of large models, while reducing reliance on a large number of graphics cards, lowering costs and improving resource utilization.

[0114] This application also provides a training and inference integrated machine, which includes a training and inference expansion card and a terminal device. The terminal device includes memory, a processor, and a training and inference unit. The training and inference expansion card interacts with the training and inference unit to exchange training and inference data. By implementing the technical solution of the above embodiments, the training and inference expansion card interacts with the training and inference unit and / or memory to improve the training and inference efficiency of large models under limited graphics card resources. The memory is the running memory of the terminal device, and the training and inference unit includes video memory and a graphics processor.

[0115] This application provides a control device for a training and pushing expansion card. The control device for the training and pushing expansion card includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the control method for the training and pushing expansion card in the first embodiment described above.

[0116] The following is for reference. Figure 4 The diagram illustrates a structural schematic of a control device suitable for implementing the training and pushing expansion card in the embodiments of this application. The control device for the training and pushing expansion card in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablets, and in-vehicle terminals, as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The control device for the training expansion card shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0117] like Figure 4 As shown, the control device of the training expansion card may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory (ROM) 1002 or a program loaded from the storage device 1003 into the random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the control device of the training expansion card. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the control device of the training expansion card to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a control device for a training expansion card with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented or possessed alternatively.

[0118] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0119] The control device for the training and pushing expansion card provided in this application, employing the control method for the training and pushing expansion card in the above embodiments, can solve the technical problem that current training and pushing schemes cannot train models with large parameters under limited graphics card resources. Compared with the prior art, the beneficial effects of the control device for the training and pushing expansion card provided in this application are the same as those of the control device for the training and pushing expansion card provided in the above embodiments, and other technical features in the control device for the training and pushing expansion card are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0120] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0121] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0122] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the control method of the training expansion card in the above embodiments.

[0123] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), or any suitable combination thereof.

[0124] The aforementioned computer-readable storage medium may be included in the control device of the training and push expansion card; or it may exist independently and not be assembled into the control device of the training and push expansion card.

[0125] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the control device of the training and push expansion card, cause the control device of the training and push expansion card to: acquire the training and push data output by the training and push unit;

[0126] Determine the video memory usage value of the training and push data;

[0127] If the video memory usage value is greater than the video memory threshold associated with the training and push unit, the training and push data is stored in the training and push expansion card.

[0128] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0131] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the control method of the training and pushing expansion card described above. This solves the technical problem that current training and pushing schemes cannot train models with large parameters under limited graphics card resources. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the control method of the training and pushing expansion card provided in the above embodiments, and will not be elaborated upon here.

[0132] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described control method for a training expansion card.

[0133] The computer program product provided in this application can solve the technical problem that current training and induction schemes cannot train models with large parameters under limited graphics card resources. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the control method for the training and induction expansion card provided in the above embodiments, and will not be repeated here.

[0134] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A control method for a training expansion card, characterized in that, The control method for the training and push expansion card includes: Obtain model parameters; The model parameters are divided into memory data and expansion card data based on the memory specifications of the training and push units. The expansion card data is stored in the training expansion card, and the video memory data is sent to the training unit; If a pre-caching instruction is received, the target data is determined from the expansion card data according to the pre-caching instruction; The target data is sent to the training and push unit; Obtain the training and induction data output by the training and induction unit, wherein the training and induction data includes model gradients, optimizer parameters, and checkpoints; Determine the video memory usage value of the training and push data; If the memory usage value is greater than the memory threshold associated with the training and push unit, the model gradient is unloaded to the training and push expansion card, the checkpoint is associated with the model parameters and stored in the training and push expansion card, and the optimizer parameters are stored in the training and push expansion card. The optimizer state is updated based on the optimizer parameters. The training and push expansion card is used to expand the memory capacity of the training and push unit, and the training and push expansion card is connected to the training and push unit through a PCIe interface for data interaction. If data to be inferred is received, the remaining video memory of the training and inference expansion card is obtained; If the remaining video memory is not lower than the operating threshold, the remaining video memory is used to execute the inference process based on the data to be inferred, so as to output the inference result.

2. The control method for the training and expansion card as described in claim 1, characterized in that, After the step of unloading the model gradient to the training and pushing extension card, the following steps are included: If a gradient update instruction is received from the training and induction unit, the gradient of the target model corresponding to the gradient update instruction is obtained; The gradient of the target model is sent to the training and pushing unit.

3. The control method for the training and expansion card as described in claim 1, characterized in that, Before the step of determining the video memory usage value of the training data, the method further includes: Determine whether the training data is used for the calculation of the next frame, where the calculation of the next frame refers to the calculation that immediately follows the current calculation step; If it is not used for calculation in the next frame, perform the step of determining the video memory usage value of the training data.

4. A training and pushing integrated machine, characterized in that, The training and push integrated machine includes: a training and push expansion card and a terminal device. The terminal device includes memory, a processor and a training and push unit. The training and push expansion card interacts with the training and push unit to exchange training and push data. The processor executes the steps of the control method of the training and push expansion card as described in any one of claims 1 to 3.

5. A control device for a training expansion card, characterized in that, The control device of the training and push expansion card includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the control method for the training and push expansion card as described in any one of claims 1 to 3.

6. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the control method for the training expansion card as described in any one of claims 1 to 3.

Images