Control method and device of training and pushing expansion card, storage medium and training and pushing all-in-one machine

Abstract

The invention discloses a training and pushing expansion card control method and device, a storage medium and a training and pushing all-in-one machine, and relates to the technical field of electronic digital data processing, and the training and pushing expansion card control method comprises the steps: obtaining training and pushing data outputted by a training and pushing unit; determining a video memory occupancy value of the training and pushing data; and if the video memory occupancy value is greater than a video memory threshold associated with the training and pushing unit, storing the training and pushing data to a training and pushing expansion card so as to reduce the video memory occupancy of the training and pushing unit, thereby solving the technical problem that a current training and pushing scheme cannot train a model with relatively large parameters under limited video card resources, and improving the training and pushing efficiency. The technical effect of large model training reasoning in a small video memory scene is realized.

Description

Technical Field

[0001] This application relates to the technical field of electronic digital data processing, and particularly to a control method, device, storage medium, and training and inference integrated machine for a training and inference expansion card. Background Art

[0002] In traditional large model local deployment and training solutions, data is first uploaded to the server cluster of cloud service providers, then model training is completed through high-performance computing nodes with multi-machine interconnection, and finally the generated model is deployed to the inference environment. However, this process highly depends on the quantity and interconnection efficiency of graphics card resources, and the training cost increases superlinearly with the model scale. Therefore, current training and inference solutions cannot train models with large parameters under limited graphics card resources. Summary of the Invention

[0003] The main objective of this application is to provide a control method, device, storage medium, and training and inference integrated machine for a training and inference expansion card, aiming to solve the technical problem that current training and inference solutions cannot train models with large parameters under limited graphics card resources.

[0004] To achieve the above objective, this application provides a control method for a training and inference expansion card, and the control method for the training and inference expansion card includes:

[0005] Obtain training and inference data output by a training and inference unit;

[0006] Determine the video memory occupancy value of the training and inference data;

[0007] If the video memory occupancy value is greater than the video memory threshold associated with the training and inference unit, store the training and inference data in the training and inference expansion card.

[0008] In an embodiment, before the step of obtaining the training and inference data output by the training and inference unit, it includes:

[0009] Obtain model parameters;

[0010] Based on the video memory specification, divide the model parameters into video memory data and expansion card data;

[0011] Store the expansion card data in the training and inference expansion card, and send the video memory data to the training and inference unit.

[0012] In an embodiment, after the step of storing the expansion card data in the training and inference expansion card and sending the video memory data to the training and inference unit, it includes:

[0013] If a pre-cache instruction is received, determine target data from the expansion card data according to the pre-cache instruction;

[0014] Send the target data to the training and inference unit.

[0015] In one embodiment, the training and inference data includes at least one of model gradients, optimizer parameters, and checkpoints. The step of storing the training and inference data in the training and inference expansion card includes at least one of the following:

[0016] Unload the model gradients to the training and inference expansion card;

[0017] Update the optimizer state based on the optimizer parameters.

[0018] In one embodiment, after the step of unloading the model gradients to the training and inference expansion card, it includes:

[0019] If a gradient update instruction from the training and inference unit is received, obtain the target model gradients corresponding to the gradient update instruction;

[0020] Send the target model gradients to the training and inference unit.

[0021] In one embodiment, the training and inference data includes model parameters. The step of storing the training and inference data in the training and inference expansion card includes:

[0022] If the training and inference data further includes a checkpoint, store the checkpoint and the model parameters in association with each other in the training and inference expansion card.

[0023] In one embodiment, before the step of determining the video memory occupancy value of the training and inference data, it further includes:

[0024] Determine whether the training and inference data is used for the next frame calculation;

[0025] If it is not used for the next frame calculation, execute the step of determining the video memory occupancy value of the training and inference data.

[0026] In one embodiment, the control method of the training and inference expansion card further includes:

[0027] If inference data to be processed is received, obtain the remaining video memory of the training and inference expansion card;

[0028] If the remaining video memory is not lower than the running threshold, call the remaining video memory to execute an inference process based on the inference data to be processed to output an inference result.

[0029] In addition, to achieve the above object, the present application further provides a training and inference integrated machine, which includes: a training and inference expansion card and a terminal device. The terminal device includes a memory, a processor, and a training and inference unit. The training and inference expansion card interacts with the training and inference unit to exchange training and inference data by implementing the control method of the training and inference expansion card as described above.

[0030] In addition, to achieve the above object, the present application further provides a control device for a training and inference expansion card. The control device for the training and inference expansion card includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the control method for the training and inference expansion card as described above.

[0031] In addition, to achieve the above object, the present application further provides a storage medium. The storage medium is a computer-readable storage medium, and a program for implementing the control method for the training and inference expansion card is stored on the computer-readable storage medium. The program for implementing the control method for the training and inference expansion card is executed by a processor to implement the steps of the control method for the training and inference expansion card as described above.

[0032] The present application provides a control method for a training and inference expansion card. The present application obtains training and inference data output by a training and inference unit, and then determines the video memory occupancy value of the training and inference data; if the video memory occupancy value is greater than the video memory threshold associated with the training and inference unit, the training and inference data is stored in the training and inference expansion card to reduce the video memory occupancy of the training and inference unit, solving the technical problem that the current training and inference solution cannot train a model with large parameters under limited video card resources, and achieving the technical effect of training and inferring a large model in a small video memory scenario.

[0033] The present application schedules the training and inference data output by the training and inference unit to dynamically allocate it to the video memory of the training and inference unit or the training and inference expansion card, so as to reduce the high-capacity demand for video memory in large model training and inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0035] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0036] Figure 1 It is a schematic flowchart provided for Embodiment 1 of the control method for the training and inference expansion card of the present application;

[0037] Figure 2 It is a schematic flowchart provided for Embodiment 2 of the control method for the training and inference expansion card of the present application;

[0038] Figure 3 It is a schematic flowchart provided for Embodiment 5 of the control method for the training and inference expansion card of the present application;

[0039] Figure 4This is a schematic diagram of the hardware structure related to the control device of the training and inference expansion card of the present application.

[0040] The implementation, functional features, and advantages of the present application will be further described in conjunction with embodiments with reference to the accompanying drawings. Detailed implementation manners

[0041] It should be understood that the specific embodiments described herein are only used to explain the technical solution of the present application and are not used to limit the present application.

[0042] To better understand the technical solution of the present application, the following will be described in detail in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0043] Currently, in traditional large model local deployment and training solutions, data is first uploaded to the server cluster of cloud service providers, then model training is completed through high-performance computing nodes with multi-machine interconnection, and finally the generated model is deployed to the inference environment. However, this process highly depends on the quantity and interconnection efficiency of graphics card resources, and the training cost increases superlinearly with the model scale. Therefore, current training and inference solutions cannot train models with large parameters under limited graphics card resources.

[0044] The main solution of the present application is to obtain the training and inference data output by the training and inference unit; determine the video memory occupancy value of the training and inference data; if the video memory occupancy value is greater than the video memory threshold associated with the training and inference unit, store the training and inference data in the training and inference expansion card. The present application schedules the training and inference data output by the training and inference unit to dynamically allocate it to the video memory of the training and inference unit or the training and inference expansion card, so as to reduce the high-capacity demand for video memory in large model training and inference.

[0045] It should be noted that the execution subject of this embodiment can be the training and inference expansion card, or a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a control device of a training and inference expansion card that can implement the above functions, or a training and inference all-in-one machine including the training and inference expansion card. This embodiment does not make specific limitations on this. The following takes the training and inference expansion card as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0046] Based on this, Embodiment 1 of the present application proposes a control method for a training and inference expansion card. Please refer to Figure 1 , and the control method of the training and inference expansion card includes steps S10 to S30:

[0047] Step S10, obtain the training and inference data output by the training and inference unit.

[0048] In this embodiment, the training and pushing unit: is a hardware device used for large model training and reasoning, covering graphics processors and dedicated acceleration chips. It can also include the video memory of the graphics processor, which is the core component for executing various computing tasks of the model. For example, the training and pushing expansion card interacts with the video memory of the training and pushing unit, or the training and pushing expansion card interacts with the graphics processor of the training and pushing unit, or the training and pushing expansion card interacts with the CUDA unit. Training and pushing data: during the model training process, it includes input training samples, model intermediate activation values, checkpoints, model gradients, model parameters, optimizer parameters, etc.; during reasoning, it includes data to be inferred, intermediate results and reasoning results, etc., and is the carrier of information in the model training and reasoning process.

[0049] As an optional implementation, a data capture mechanism is set at the output interface of the training and pushing unit. For training and reasoning using a deep learning framework, the callback function or hook mechanism provided by the framework can be used to capture the intermediate results as training and pushing data after a specific calculation step is completed. For hardware devices, the training and pushing data can be obtained through the underlying driver. For example, NVIDIA GPU can use the CUDA programming interface to read relevant data from the video memory after the GPU kernel function is executed.

[0050] Step S20, determining the video memory occupancy value of the training and pushing data.

[0051] In this embodiment, the video memory occupancy value refers to the storage space occupied by the training and pushing data in the video memory of the training and pushing unit, which is measured in bytes and reflects the physical space occupancy of the training and pushing data in the video memory.

[0052] As an optional implementation, for different types of data, the video memory occupancy value can be calculated based on its data type and dimension. In the deep learning framework, the element size and total number of elements of the tensor type training data can be obtained through the method provided by the framework, and then the video memory occupancy value can be calculated.

[0053] Step S30: If the video memory occupancy value is greater than the video memory threshold associated with the training and push unit, the training and push data is stored in a training and push expansion card.

[0054] In this embodiment, the video memory threshold is a quantitative limit set for each training and inference data unit. Since in addition to training and inference data, other information such as model parameters is also stored in the video memory, the training and inference data usually does not directly exceed the total upper limit of the video memory, but there is a set value specifically for the training and inference data. And the video memory threshold is associated with the training and inference unit, that is, the video memory threshold is positively correlated with the video memory size of the training and inference unit. Training and inference expansion card: It is a hardware device connected to the training and inference unit through a high-speed interface, and its main function is to expand the video memory capacity of the training and inference unit. When the video memory occupancy value of the training and inference data exceeds the video memory threshold, the training and inference expansion card can provide additional storage space to store this training and inference data, thereby reducing the pressure on the video memory of the training and inference unit.

[0055] As an alternative implementation, the video memory threshold is set according to the performance of the training and inference unit, the characteristics of the model, and the actual requirements of the system. When the video memory occupancy value of the training and inference data calculated by the main control is greater than the set video memory threshold, the main control stores the training and inference data in the corresponding mount point directory of the training and inference expansion card in a specific file format through the file system interface of the operating system or a dedicated hardware driver interface.

[0056] Through the above steps, the present application realizes the dynamic scheduling of the training and inference data output by the training and inference unit, enabling it to be reasonably allocated to the video memory of the training and inference unit or the training and inference expansion card according to the usage of the video memory and the video memory threshold, thereby effectively reducing the high-capacity requirement of the video memory for large model training and inference.

[0057] Based on Embodiment 1, Embodiment 2 of the present application proposes a control method for a training and inference expansion card. Before step S10, it includes:

[0058] Step A10, obtaining model parameters.

[0059] In this embodiment, model parameters: In a large model, model parameters are a set of numerical values that determine the function and performance of the model. For example, in a neural network, model parameters include connection weights between neurons, bias values, etc. These parameters are continuously adjusted during the training process so that the model can learn the patterns and rules in the data, thereby achieving accurate prediction or classification of new data.

[0060] As an alternative implementation, the model parameters are read from a storage device, the Internet, or cloud storage. Model parameters are usually stored in the form of files on a local hard disk, network storage device, or cloud storage. These storage devices can be accessed through a file system interface or a network interface to read the model parameter files. For example, in common deep learning frameworks, the model parameter files may be saved in formats such as.pth (PyTorch),.ckpt (TensorFlow), etc.

[0061] As another alternative implementation, in a distributed training environment, model parameters are synchronously obtained from multiple nodes. A distributed communication protocol (such as MPI, NCCL, etc.) is used to achieve parameter synchronization between nodes.

[0062] Step A20: Based on the video memory specification, divide the model parameters into video memory data and expansion card data.

[0063] In this embodiment, the video memory specification refers to the physical characteristics such as the capacity size and bandwidth of the video memory of the training and inference unit. The video memory specification determines the amount of data that the video memory can store and process, as well as the data transmission speed. The video memory data is a part of the data divided from the model parameters, and the size of this part of the data is within the capacity that the video memory of the training and inference unit can accommodate, and can be directly stored in the video memory for the training and inference unit to perform fast calculations and processing. The expansion card data is also divided from the model parameters. Since its size exceeds the capacity that the video memory of the training and inference unit can accommodate, or in order to optimize the use efficiency of the video memory, this part of the data is stored in the training and inference expansion card.

[0064] As an alternative implementation, by querying the hardware information or relevant interfaces provided by the operating system, obtain the total capacity of the video memory of the training and inference unit and the currently occupied capacity, and then calculate the available capacity of the video memory. Based on the overall scale of the model parameters and the available capacity of the video memory, divide the model parameters. Determine the part of the model parameters within the available capacity of the video memory as the video memory data, and the remaining model parameters that exceed the available capacity of the video memory are classified as expansion card data.

[0065] Exemplarily, if the available capacity of the video memory is 4GB and the total size of the model parameters is 10GB, then the first 4GB of the model parameters are used as the video memory data, and the latter 6GB are used as the expansion card data.

[0066] Exemplarily, if the available capacity of the video memory is 16GB and the total size of the model parameters is 648GB, then the first 16GB of the model parameters are used as the video memory data, and the latter 632GB are used as the expansion card data.

[0067] Step A30: Store the expansion card data in the training and inference expansion card, and send the video memory data to the training and inference unit.

[0068] In this embodiment, the expansion card data is saved to the storage chip of the training and inference expansion card so that it can be read and used when needed later. The video memory data is transferred from the storage device or other nodes to the video memory of the training and inference unit, enabling the training and inference unit to directly access and process this data.

[0069] As an alternative implementation, using the interface provided by the training and inference expansion card (such as the PCIe interface), the expansion card data is written into the storage medium of the training and inference expansion card. During the writing process, operations such as data formatting and verification are performed to ensure the integrity and correctness of the data during storage. Through a high-speed data transmission interface (such as the GPU bus), the video memory data is transferred from the storage device or other nodes to the video memory of the training and inference unit. During the transmission process, the order and rate of data transmission are reasonably planned to make full use of the data transmission bandwidth and improve the transmission efficiency.

[0070] Through the above steps, the model parameters are reasonably processed and allocated at the start of training, providing a good data foundation for the subsequent training and inference processes. Combining with the subsequent training and inference data processing steps, it can effectively optimize the use of video memory in large model training and inference and reduce the demand for high-capacity video memory.

[0071] Optionally, after step A30, it includes:

[0072] Step A40, if a pre-cache instruction is received, determine the target data in the expansion card data according to the pre-cache instruction.

[0073] In this embodiment, the pre-cache instruction: an instruction issued by the system for indicating that certain data needs to be loaded from the training and inference expansion card to the video memory of the training and inference unit in advance. This instruction is usually generated based on the analysis and prediction of the subsequent calculation tasks of the training and inference unit, aiming to reduce the data waiting time during the training and inference process and improve the calculation efficiency. The target data: the model parameter data that is filtered out from the expansion card data according to the pre-cache instruction and is required by the training and inference unit during the next calculation process.

[0074] As an alternative implementation, after receiving the pre-cache instruction, the pre-cache instruction is parsed to extract the identification information about the required model parameters contained therein. Based on this identification information, a search and match are performed in the expansion card data stored in the training and inference expansion card to determine the target data. For example, if the pre-cache instruction indicates that the weight parameters of a certain layer of the neural network are required, then the weight parameters of this layer are located in the expansion card data as the target data.

[0075] Step A50, send the target data to the training and inference unit.

[0076] In this embodiment, using the data transmission interface (such as the PCIe interface) between the training and inference expansion card and the training and inference unit, the determined target data is read out from the storage medium of the training and inference expansion card and sent to the video memory of the training and inference unit. During the transmission process, a similar method to that of transmitting video memory data is adopted to optimize the data transmission, such as reasonably arranging the transmission order and controlling the transmission rate, etc., to improve the efficiency and reliability of data transmission.

[0077] Through the above steps, the model parameters are reasonably processed and allocated at the beginning of training, and subsequently, according to the pre-caching instructions, the required expansion card data is loaded into the video memory of the training and inference unit, providing the necessary data support for the calculation of the training and inference unit. Combining with the subsequent training and inference data processing steps, it can effectively optimize the use of video memory for large model training and inference, reduce the demand for high-capacity video memory, and improve the overall training and inference efficiency.

[0078] Based on any of the above embodiments, Embodiment 3 of the present application proposes a control method for a training and inference expansion card. The step of storing the training and inference data in the training and inference expansion card includes at least one of the following:

[0079] Step S31, unloading the model gradient to the training and inference expansion card.

[0080] In this embodiment, the training and inference data includes at least one of model gradient, optimizer parameters, and checkpoints. Model gradient: In large model training, the model gradient is the partial derivative of the objective function with respect to the model parameters. It represents the rate of change of the objective function at the current parameter values, reflects in which direction the model parameters should be adjusted to reduce the objective function value, and is an important basis for the model to update parameters. Optimizer parameters: The optimizer is used to adjust the model parameters to minimize the loss function. The optimizer parameters are the parameters used by the optimizer during the update process, such as the learning rate, momentum, etc. Different optimizers have different parameter settings, and these parameters will affect the training speed and effect of the model. Checkpoint: It is a snapshot of the model during training, containing the state of the model at a specific moment, such as the parameter values of the model, the state of the optimizer, etc. Saving checkpoints can facilitate resuming training after training is interrupted and can also be used for model evaluation and comparison. Unloading: The process of moving data from the video memory of the training and inference unit to the training and inference expansion card, aiming to release the space of the video memory of the training and inference unit to cope with the situation of insufficient video memory. Optimizer state: The internal state of the optimizer during training. For example, when using the Stochastic Gradient Descent (SGD) optimizer, the accumulated value of momentum is part of the optimizer state. The optimizer state will be continuously updated during training to guide the update of model parameters.

[0081] As an alternative implementation, when the training and inference data output by the training and inference unit contains model gradients and the video memory occupancy value of the calculated model gradients is greater than the video memory threshold associated with the training and inference unit, the following operations are performed: Data tagging: Add a unique identifier to the model gradient data, where the identifier contains information such as data type (model gradient), generation time, and the model layer it belongs to, facilitating subsequent management and search in the training and inference expansion card. Data transmission: Transmit the model gradient data from the video memory of the training and inference unit to the storage medium of the training and inference expansion card through a high-speed interface (such as a PCIe interface). During the transmission process, use a data compression algorithm (such as Huffman coding) to compress the model gradient data to reduce the data transmission volume and transmission time. Storage management: Allocate storage space for the model gradient data in the training and inference expansion card, and record its storage location and relevant metadata (such as identifier, size, etc.). The storage management can be carried out in the form of a file system or a database to facilitate quick location and access to the model gradient data.

[0082] Step S32, update the optimizer state based on the optimizer parameters.

[0083] As an alternative implementation, when the training and inference data output by the training and inference unit contains optimizer parameters, the following operations are performed: Parameter parsing: Parse the optimizer parameters to extract the values of each parameter, such as the learning rate, momentum, etc. Different optimizers have different parameter formats, and parsing needs to be carried out according to the specific optimizer type. State update: Update the internal state of the optimizer according to the parsed optimizer parameters. For example, when using a stochastic gradient descent optimizer with momentum, update the cumulative value of the momentum according to the new momentum parameter. The update process needs to follow the update rules of the optimizer to ensure the correctness of the optimizer state. State saving: Save the updated optimizer state to the video memory of the training and inference unit or the training and inference expansion card. If the video memory space is sufficient, the optimizer state can be saved to the video memory for quick access; if the video memory space is tight, the optimizer state is saved to the training and inference expansion card, and its storage location is recorded.

[0084] Exemplarily, the training and inference expansion card is connected to the GPU through a PCIe interface. During the training process, the video memory space is limited, and the training and inference data needs to be reasonably managed.

[0085] The model is trained using a training dataset. In each training step, the loss function of the model is calculated, and the model gradients are calculated through the backpropagation algorithm. Video memory occupancy calculation: Calculate the video memory occupancy value of the model gradients. Assuming that the video memory threshold associated with the training and inference unit is 4GB, when the video memory occupancy value of the model gradients exceeds 4GB, a data offloading operation is triggered. Add an identifier to the model gradient data, such as "gradient_20250226_1000_layer3", indicating that this is the model gradient of the 3rd layer generated at 10:00 on February 26, 2025. Compress the model gradient data using Huffman coding and transfer the compressed data to the training and inference expansion card through the PCIe interface. Create a folder "gradients" in the file system of the training and inference expansion card, store the model gradient data in this folder, and record its storage path and related meta-information. The Stochastic Gradient Descent optimizer with momentum is used. In each training step, the learning rate and momentum parameters of the optimizer are updated. Parse the optimizer parameters and update the internal state of the optimizer according to the new parameter values, such as updating the cumulative value of the momentum. Due to the tight video memory space, the updated optimizer state is saved to the training and inference expansion card, and the storage location of the optimizer state in the training and inference expansion card is recorded in the video memory of the training and inference unit.

[0086] Through the above embodiments, in an actual model training scenario, the control method of the training and inference expansion card is applied to reasonably store and manage the training and inference data, so as to reduce the high-capacity requirement of video memory for large model training.

[0087] Optionally, after step S31, it further includes:

[0088] Step S311, if a gradient update instruction from the training and inference unit is received, obtain the target model gradient corresponding to the gradient update instruction;

[0089] Step S312, send the target model gradient to the training and inference unit.

[0090] In this embodiment, the gradient update instruction: An instruction sent by the training and inference unit for requesting to obtain a specific model gradient for gradient update operations. This instruction contains relevant identification information of the required model gradient, such as generation time, model layer belonging to, etc., so as to accurately locate the target model gradient. Target model gradient: The model gradient data that is found and determined from the training and inference expansion card according to the gradient update instruction and is required by the training and inference unit during the gradient update process. Cumulative gradient: In some deep learning training strategies, in order to simulate a larger batch size, the model gradients are gradually accumulated in multiple training steps, and then the accumulated gradients are used to update the model parameters after reaching a certain number of steps.

[0091] As an alternative implementation, the training and inference expansion card listens to the instructions from the training and inference unit in real time. When receiving a gradient update instruction, it parses the instruction and extracts the identification information of the target model gradient contained therein, such as the generation time, the model layer to which it belongs, etc. According to the parsed identification information, it searches in the storage management system of the training and inference expansion card. When the training and inference expansion card stores the model gradient data, it will record the relevant meta-information of each data (such as identification, storage location, etc.), and through these meta-information, the storage location of the target model gradient can be quickly located. After finding the target model gradient data, it verifies it to ensure the integrity and correctness of the data. Verification can be performed through methods such as checksum and hash value. If the data verification fails, corresponding error handling measures need to be taken, such as re-obtaining the data or reporting error information. Read the target model gradient data from the storage medium of the training and inference expansion card. If the data was compressed during storage, such as using Huffman coding, then decompression operation needs to be performed after reading the data. Send the read and decompressed target model gradient data to the video memory of the training and inference unit through a high-speed interface. During the transmission process, some optimization strategies can be adopted, such as data block transmission, parallel transmission, etc., to improve the transmission efficiency.

[0092] Exemplarily, the TensorFlow framework is used to train a recurrent neural network (RNN) model for text generation tasks. The training and inference unit is an AMD GPU, and the training and inference expansion card is connected to the GPU through a PCIe 4.0 interface. During the training process, to reduce the video memory pressure, the model gradients are offloaded to the training and inference expansion card. At the same time, a training strategy of accumulating gradients is adopted, and the model parameters are updated every 4 training steps. In each training step, the training and inference unit calculates the loss function of the model and calculates the model gradients through the backpropagation algorithm. When the video memory occupancy value of the calculated model gradients exceeds the video memory threshold associated with the training and inference unit, the model gradients are offloaded to the training and inference expansion card. A unique identifier is added to each model gradient data, such as "gradient_20250226_1400_layer5", indicating that this is the model gradient of the 5th layer generated at 14:00 on February 26, 2025. After every 4 training steps are completed, the training and inference unit needs to update the model parameters, and at this time, a gradient update instruction is issued. The instruction contains the identifier information of all the model gradients generated in these 4 training steps. After receiving the gradient update instruction, the training and inference expansion card parses the instruction and extracts the identifier information of the 4 model gradients. According to the identifier information, the corresponding model gradient data files are searched in the file system of the training and inference expansion card. Each model gradient data file is stored in a folder named "gradients" and classified and stored according to the identifier information. The 4 model gradient data files are read from the storage medium of the training and inference expansion card and decompressed. The decompressed model gradient data is transferred to the video memory of the training and inference unit in parallel in chunks through the PCIe 4.0 interface. After receiving all the model gradient data, the training and inference unit sends a confirmation message to the training and inference expansion card to complete the data transfer task required for this gradient update. Gradient accumulation and model update: The training and inference unit accumulates the 4 received model gradients in the video memory, and then uses the accumulated gradients to update the model parameters and continue the next round of training.

[0093] Through the above embodiments, the target model gradients are obtained from the training and inference expansion card according to the gradient update instruction and sent to the training and inference unit to support the training strategy of accumulating gradients, and at the same time, the high-capacity requirement of the video memory for large model training is effectively reduced.

[0094] Based on any of the above embodiments, Embodiment 4 of the present application proposes a control method for a training and inference expansion card. The step of storing the training and inference data in the training and inference expansion card includes:

[0095] Step S33, if the training and inference data further includes a checkpoint, the checkpoint is associated with the model parameters and stored in the training and inference expansion card.

[0096] In this embodiment, the training and inference data includes model parameters.

[0097] As an alternative implementation, the checkpoint and model parameters are stored in the training and inference expansion card in a correlated manner, so that when they are needed in the future, the corresponding model parameters can be quickly located according to the checkpoint, or the corresponding checkpoint information can be found according to the model parameters.

[0098] Exemplarily, the training and inference unit monitors the computing tasks of the model in real time and analyzes whether each model parameter in the current video memory will be used in the next computing step. The judgment can be made by tracking the computing graph of the model, analyzing the training and inference processes, etc. For example, during the training of some deep neural networks, the parameters of certain layers may not need to participate in the calculation temporarily at a specific training stage, and the training and inference unit can identify these parameters. When it is determined that there are model parameters in the video memory that do not need to participate in the calculation, these model parameters are transferred from the video memory of the training and inference unit to the training and inference expansion card. A unique identifier is added to the model parameters to be unloaded, and the identifier contains information such as the model layer to which the parameters belong and the generation time, which is convenient for subsequent management and search in the training and inference expansion card. The model parameters are transferred from the video memory of the training and inference unit to the storage medium of the training and inference expansion card using a high-speed interface. During the transfer process, the model parameters are compressed to reduce the data transfer volume and transfer time. Storage space is allocated for the model parameters in the training and inference expansion card, and their storage locations and related meta-information are recorded. The storage management can be carried out in the form of a database or a file system to facilitate the quick location and access of the model parameters. Before storing the model parameters and the checkpoint, an association relationship between them is established. This can be achieved by creating an association table in the database or adopting a specific naming rule in the file system. For example, an association table is created in the database to record the identifier of the model parameters and the identifier of the corresponding checkpoint; in the file system, the checkpoint file and the model parameter file can be named with the same prefix to indicate their association relationship. The relevant meta-information is updated in the storage management system of the training and inference expansion card, including the storage locations of the checkpoint and the model parameters, the association relationship, etc. In this way, when needed in the future, the relevant data can be quickly located and obtained according to the meta-information.

[0099] Exemplarily, during the training process, due to limited video memory space, it is necessary to reasonably manage the model parameters. At the same time, to facilitate the recovery of training and model evaluation, checkpoints are saved regularly. The language model is trained using the training dataset. In each training step, the training and inference unit analyzes whether each model parameter in the current video memory needs to participate in the calculation in the next calculation step. For example, when performing the calculation of the multi-head attention mechanism, it is found that the feed-forward neural network parameters of some layers are not needed in the current step. Add an identifier to these model parameters that do not need to participate in the calculation, such as "param_20250226_1600_layer8", indicating that this is the model parameter of the 8th layer generated at 16:00 on February 26, 2025. Use the LZ77 compression algorithm to compress the model parameters, and transfer the compressed data to the training and inference expansion card through the PCIe 5.0 interface. Create a folder named "parameters" in the file system of the training and inference expansion card, store the model parameter files in this folder, and record their storage paths and related meta-information. Save a checkpoint every 100 training steps. At the end of the 100th step, save the checkpoint of the current model, including information such as model parameter values and optimizer states. Add an identifier to the checkpoint, such as "checkpoint_20250226_1600_step100". Create an association table in the database to record the association relationship between "param_20250226_1600_layer8" and "checkpoint_20250226_1600_step100". Create a folder named "checkpoints" in the file system of the training and inference expansion card, store the checkpoint files in this folder, and record their storage paths and related meta-information. Update the meta-information in the storage management system of the training and inference expansion card, including the storage locations of checkpoints and model parameters, association relationships, etc.

[0100] Through the above embodiments, it is shown how to unload the model parameters that do not need to participate in the calculation to the training and inference expansion card and store the checkpoints in association with the model parameters in an actual model training scenario, thereby effectively reducing the high-capacity requirement of video memory for large model training and facilitating the recovery of training and model evaluation.

[0101] Based on any of the above embodiments, Embodiment 5 of the present application proposes a control method for a training and inference expansion card. Before step S20, it includes: determining whether the training and inference data is used for the next frame calculation; if it is not used for the next frame calculation, execute the step of determining the video memory occupancy value of the training and inference data.

[0102] In this embodiment, next-frame calculation: calculation steps that are sequentially performed in chronological order during the training or inference of the model. Next-frame calculation refers to the calculation step immediately following the current calculation step. Whether the training and inference data is used for the next-frame calculation determines whether the data needs to be immediately retained in the video memory of the training and inference unit to meet the fast access requirement.

[0103] As an alternative implementation, the computational graph of the large model is extremely complex, containing thousands or even tens of thousands of nodes and edges. Advanced graph analysis algorithms are used to deeply analyze the computational graph to clarify the dependency relationships and usage locations of each training and inference data in the graph. For example, in a large language model based on the Transformer architecture, the output data of certain intermediate layers may only be used in specific multi-head attention mechanisms or feed-forward network modules. By analyzing the computational graph, it can be accurately determined whether this data will be used in the next frame of calculation. During the training and inference processes of the large model, track the computational progress and status in real-time and accurately. Use timestamps and event logs in the distributed system to record the generation time and usage time of each training and inference data, and combine the current computational steps to accurately determine whether this data will be used in the next step of calculation. For example, in the distributed training of a large model, the intermediate gradient data generated by some nodes may only be used for local updates in the current iteration step and will not be used in the next frame of calculation. When it is determined that the training and inference data is not used in the next frame of calculation, perform the following operations to determine its video memory occupancy value: The data types in the large model are diverse and complex. In addition to common integers, floating-point numbers, and boolean values, there may also be custom data types. Special data type recognition algorithms are used to accurately recognize the data types of the training and inference data. For example, for the quantized data in the large model, accurately recognize its quantization bits and data format to determine the storage space occupied by each element. The data in the large model usually has extremely high dimensions. For example, the dimensions of a tensor may reach dozens or even hundreds. Efficient dimension calculation algorithms are used to accurately calculate the sizes of each dimension of the multi-dimensional data. By multiplying the number of bytes occupied by the data type by the total number of elements of the data (i.e., the product of the sizes of each dimension), the video memory occupancy value of the training and inference data is obtained. For example, for a 16-bit floating-point tensor with a shape of (1024, 1024, 1024, 64), the total number of elements is 1024×1024×1024×64, and each element occupies 2 bytes. Therefore, the video memory occupancy value of this tensor is 1024×1024×1024×64×2 bytes. When the video memory occupancy value of the training and inference data is greater than the video memory threshold, perform the following operations to store the training and inference data in the training and inference expansion card: Add a unique and detailed identifier to the training and inference data. The identifier contains rich information such as the data type, generation time, model layer to which it belongs, and data source node. At the same time, construct an efficient data index structure to facilitate subsequent rapid management and search in the training and inference expansion card. For example, use data structures such as hash tables or B-trees to store the mapping relationship between the data identifier and the storage location. Transfer the training and inference data from the video memory of the training and inference unit to the storage medium of the training and inference expansion card through a high-speed and low-latency interface (such as a PCIe 5.0 or higher-level interface). Use advanced data compression algorithms (such as the Zstd compression algorithm) to efficiently compress the training and inference data to significantly reduce the data transfer volume and transfer time. At the same time, use multi-channel parallel transfer technology to further improve the transfer efficiency.Due to the huge amount of training and inference data of large models, the training and inference expansion cards may adopt a distributed storage architecture. When storing training and inference data, a distributed file system (such as Ceph, GlusterFS, etc.) is used for management, and the data is scattered and stored on multiple storage nodes. At the same time, record the storage location and related meta-information of each data block (such as identification, size, checksum, etc.) to ensure the reliability and accessibility of the data.

[0104] Exemplarily, DeepSpeed framework is being used to train a large language model with 1000B parameters. The training and inference unit is a cluster composed of 64 NVIDIA A100 GPUs. The training and inference expansion card is connected to the GPUs through a high-speed PCIe 5.0 interface and adopts a distributed storage architecture. In each training step, through the customized hook mechanism of the DeepSpeed framework, capture the output data of each layer of the model as training and inference data. For example, add data collection code in the forward propagation methods of the self-attention layer and the feed-forward network layer of the model to collect the output results of each layer. Use graph analysis algorithms to deeply analyze the computational graph of the large language model and find that the output data of some intermediate layers is no longer used after the subsequent layer normalization operation, that is, not used for the next frame calculation. Real-time track the computational status of each GPU node in the cluster, record the generation time and usage time of each training and inference data, and further confirm that these data will not be used in the next calculation. For the training and inference data determined not to be used for the next frame calculation, accurately identify its data type as 16-bit floating-point number and calculate its dimension size. Suppose the output tensor shape of a certain intermediate layer is (2048, 4096, 1024), then the total number of its elements is 2048×4096×1024, and each element occupies 2 bytes. Therefore, the video memory occupancy value of this tensor is 2048×4096×1024×2 bytes. If the video memory occupancy value is greater than the video memory threshold associated with the training and inference unit, store this training and inference data in the training and inference expansion card. Add a detailed identification "output_20250226_2000_layer5_node12" to this training and inference data, indicating that this is the output data of the 5th layer and from the 12th node generated at 20:00 on February 26, 2025. At the same time, add this identification to the data index structure. Use the Zstd compression algorithm to efficiently compress this training and inference data, and through the multi-channel parallel transmission technology of the PCIe 5.0 interface, quickly transmit the compressed data to the distributed storage system of the training and inference expansion card. Allocate storage space for this training and inference data in the distributed file system, scatter the data blocks and store them on multiple storage nodes, and record the storage location and related meta-information of each data block.

[0105] Through the above embodiments, it is determined whether the training and inference data is used for the next frame calculation, and the unused data is stored in the training and inference expansion card according to the video memory occupancy value and the video memory threshold, effectively reducing the high-capacity requirement of the large model training for the video memory and improving the training efficiency and resource utilization rate.

[0106] Based on any of the above embodiments, Embodiment Six of the present application proposes a control method for a training and inference expansion card, further including:

[0107] Step B10, if the data to be inferred is received, obtain the remaining video memory of the training and inference expansion card.

[0108] In this embodiment, the data to be inferred: In the real-time training (training while inferring) scenario, the data that needs to be input into the large model for inference calculation. For example, in a natural language processing task, the data to be inferred may be a piece of text; in an image recognition task, it may be an image. The remaining video memory of the training and inference expansion card: The size of the unused storage space in the current video memory of the training and inference expansion card. Since the training and inference expansion card is used to store part of the training and inference data to relieve the video memory pressure of the training and inference unit, its remaining video memory reflects the available resources currently available for executing the inference process. The running threshold: A minimum available space standard set for the video memory of the training and inference expansion card. When the remaining video memory is not lower than this threshold, it indicates that the training and inference expansion card has sufficient resources to execute the inference process; otherwise, the inference may not be able to proceed normally.

[0109] Step B20, if the remaining video memory is not lower than the running threshold, call the remaining video memory to execute an inference process based on the data to be inferred to output an inference result.

[0110] In this embodiment, the inference process: The process in which the large model calculates based on the input data to be inferred using the learned model parameters to obtain an inference result. In the real-time training scenario, the inference process and the training process are carried out in parallel.

[0111] As an alternative implementation, the training and inference unit interacts with the training and inference expansion card through a specific communication protocol (such as the PCIe interface communication protocol). The training and inference unit sends a request instruction to the training and inference expansion card to obtain the remaining video memory. After receiving the request instruction, the training and inference expansion card queries the usage of the video memory through its internal hardware management module. This module can directly access the status register of the video memory to obtain information on the used video memory space and the total video memory space. Based on the queried used video memory space and total video memory space, the remaining video memory of the training and inference expansion card is calculated. For example, if the total video memory of the training and inference expansion card is 1024GB and 20GB has been used, the remaining video memory is 1004GB. When the remaining video memory of the training and inference expansion card is not lower than the running threshold, the data to be inferred is loaded from the input device (such as a network interface, storage device, etc.) into the video memory of the training and inference expansion card. During the loading process, it may be necessary to preprocess the data, such as operations like normalization, cropping, encoding, etc., to make it meet the input requirements of the large model. The relevant parameters of the large model are obtained from the storage of the training and inference expansion card and loaded into the video memory. Due to the huge number of parameters in the large model, techniques such as parameter partitioning and asynchronous loading may be adopted to improve the loading efficiency. The training and inference expansion card calls its computing core (such as a GPU core, a dedicated inference chip, etc.) to perform inference calculations based on the loaded data to be inferred and the model parameters. During the calculation process, parallel computing techniques are used to accelerate the inference speed, such as multi-threaded computing, SIMD instruction sets, etc. After the inference calculation is completed, the obtained inference result is output from the video memory of the training and inference expansion card. The result can be returned to the requesting end (such as a client, an application program, etc.), or further processed and analyzed.

[0112] Exemplarily, a real-time training and inference task of an extremely large-scale natural language processing large model is in progress, and the number of parameters of this model is as high as 1000B. Originally, the traditional method required 32 NVIDIA A100 graphics cards to meet the training and inference requirements. Now, with the training and inference expansion card, only 4 NVIDIA A100 graphics cards are needed as the training and inference units. The training and inference expansion card is connected to the training and inference units through a high-speed PCIe 5.0 interface, and the running threshold is set to 50GB. The natural language processing system receives a large amount of text data input by users as the data to be inferred, and these text data cover various fields of question consultation and sentence understanding requirements. The training and inference units send a request instruction to obtain the remaining video memory to the training and inference expansion card through the PCIe 5.0 interface. The hardware management module of the training and inference expansion card queries the video memory status register and learns that the total video memory is 512GB, 220GB has been used, and the remaining video memory is calculated to be 292GB. Since the remaining video memory of 292GB is not lower than the running threshold of 50GB, the training and inference expansion card starts to execute the inference process. The text data to be inferred is quickly loaded from the high-speed network interface into the video memory of the training and inference expansion card using multi-thread parallel technology, and at the same time, preprocessing operations such as word segmentation, part-of-speech tagging, and syntactic analysis are performed on the data. The parameter block storage and asynchronous loading technology are adopted to quickly load the relevant parameters of the large language model from the large-capacity storage of the training and inference expansion card into the video memory. The training and inference expansion card calls its multi-core GPU cores and dedicated inference chip array, and based on the loaded text data and model parameters, uses the CUDA parallel programming model and tensor parallel computing technology to perform large-scale inference calculations to accelerate the inference process. After the inference calculation is completed, the generated accurate response text and semantic understanding results are output from the video memory of the training and inference expansion card as the inference results and returned to the user to meet the user's consultation and understanding requirements.

[0113] Through the above embodiments, in the scenario of a large model with real-time training (training while inferring), it is judged whether to execute the inference process based on the remaining video memory of the training and inference expansion card, giving full play to the advantage of the large capacity of the training and inference expansion card, achieving efficient inference of the large model, while reducing the dependence on a large number of graphics cards, reducing costs and improving resource utilization.

[0114] The embodiment of the present application also provides a training and inference all-in-one machine, which includes: a training and inference expansion card and a terminal device. The terminal device includes a memory, a processor, and a training and inference unit. The training and inference expansion card interacts with the training and inference unit for training and inference data. The training and inference expansion card interacts with the training and inference unit and / or the memory by implementing the technical solutions of the above embodiments to improve the training and inference efficiency of the large model under limited graphics card resources. Among them, the memory is the operating memory of the terminal device, and the training and inference unit includes a video memory and a graphics processor.

[0115] The present application provides a control device for a training and inference expansion card. The control device for the training and inference expansion card includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the control method for the training and inference expansion card in Embodiment 1 above.

[0116] Reference is made below Figure 4 to FIG. , which shows a schematic structural diagram of a control device for a training and inference expansion card suitable for implementing the embodiments of the present application. The control device for the training and inference expansion card in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, vehicle-mounted terminals, etc., and fixed terminals such as digital TVs, desktop computers, and the like. Figure 4 The control device for the training and inference expansion card shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0117] As Figure 4 shown, the control device for the training and inference expansion card may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may execute various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the random access memory 1004, various programs and data required for the operation of the control device for the training and inference expansion card are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the control device for the training and inference expansion card to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a control device for the training and inference expansion card having various systems, it should be understood that it is not required to implement or include all the shown systems. More or fewer systems may be alternatively implemented or included.

[0118] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0119] The control device of the training and inference expansion card provided by the present application adopts the control method of the training and inference expansion card in the above embodiment, and can solve the technical problem that the current training and inference solution cannot train a model with large parameters under limited graphics card resources. Compared with the prior art, the beneficial effects of the control device of the training and inference expansion card provided by the present application are the same as those of the control device of the training and inference expansion card provided by the above embodiment, and other technical features in the control device of the training and inference expansion card are the same as the features disclosed in the method of the previous embodiment, and will not be elaborated here.

[0120] It should be understood that each part disclosed in the present application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0121] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0122] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the control method of the training and inference expansion card in the above embodiment.

[0123] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination of the above.

[0124] The above computer-readable storage medium can be included in the control device of the training and inference expansion card; it can also exist separately and not be assembled into the control device of the training and inference expansion card.

[0125] The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the control device of the training and inference expansion card, the control device of the training and inference expansion card is caused to: obtain the training and inference data output by the training and inference unit;

[0126] determine the video memory occupancy value of the training and inference data;

[0127] If the video memory occupancy value is greater than the video memory threshold associated with the training and inference unit, store the training and inference data in the training and inference expansion card.

[0128] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and this module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0130] The modules described in the embodiments of this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0131] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for performing the control method of the above-mentioned training and inference expansion card, and can solve the technical problem that the current training and inference solution cannot train models with larger parameters under limited graphics card resources. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the control method of the training and inference expansion card provided in the above embodiments, and will not be elaborated here.

[0132] An embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the control method of the training and inference expansion card as described above.

[0133] The computer program product provided by the present application can solve the technical problem that the current training and inference solution cannot train a model with large parameters under limited graphics card resources. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as those of the control method of the training and inference expansion card provided by the above embodiment, and will not be elaborated here.

[0134] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied to other related technical fields, shall be included in the patent scope of the present application by the same token.

Claims

1. A control method for a training and pushing expansion card, characterized in that: The control method of the training and pushing expansion card comprises: Obtaining training and pushing data output by the training and pushing unit; Determine the video memory occupancy value of the training and pushing data; If the video memory occupancy value is greater than a video memory threshold associated with the training and pushing unit, the training and pushing data is stored in a training and pushing expansion card.

2. The control method of the training expansion card as claimed in claim 1, characterized in that: Before the step of obtaining the training and pushing data output by the training and pushing unit, the method includes: Get model parameters; Splitting the model parameters into video memory data and expansion card data based on video memory specifications; The expansion card data is stored in the training-pushing expansion card, and the video memory data is sent to the training-pushing unit.

3. The control method of the training expansion card as claimed in claim 2, characterized in that: After the step of storing the expansion card data in the training-pushing expansion card and sending the video memory data to the training-pushing unit, the method further includes: If a pre-caching instruction is received, determining target data in the expansion card data according to the pre-caching instruction; The target data is sent to the training and derivation unit.

4. The control method of the training expansion card as claimed in claim 1, characterized in that: The training data includes at least one of a model gradient, an optimizer parameter, and a checkpoint. The step of storing the training data in the training expansion card includes at least one of the following: Unloading the model gradient to the training and pushing expansion card; An optimizer state is updated based on the optimizer parameters.

5. The control method of the training expansion card as claimed in claim 4, characterized in that: After the step of unloading the model gradient to the training expansion card, the method further comprises: If a gradient update instruction of the training and pushing unit is received, a target model gradient corresponding to the gradient update instruction is obtained; The target model gradient is sent to the training and pushing unit.

6. The control method of the training expansion card as claimed in claim 1, characterized in that: The training data includes model parameters, and the step of storing the training data in the training expansion card includes: If the training data further includes a checkpoint, the checkpoint is associated with the model parameter and stored in the training expansion card.

7. The control method of the training expansion card as claimed in claim 1, characterized in that: Before the step of determining the video memory occupancy value of the training and pushing data, the method further includes: Determining whether the training and pushing data is used for next frame calculation; If it is not used for the next frame calculation, the step of determining the video memory occupancy value of the training and pushing data is performed.

8. The control method of the training expansion card as claimed in claim 1, characterized in that: The control method of the training and pushing expansion card also includes: If the data to be inferred is received, obtaining the remaining video memory of the training and inference expansion card; If the remaining amount of the video memory is not lower than the operating threshold, the remaining amount of the video memory is called to execute an inference process based on the data to be inferred, so as to output an inference result.

9. A training and pushing machine, characterized in that: The training-push integrated machine includes: a training-push expansion card and a terminal device, the terminal device includes a memory, a processor and a training-push unit, and the training-push expansion card exchanges training-push data with the training-push unit.

10. A control device for a training and pushing expansion card, characterized in that: The control device of the training-pushing expansion card includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the control method of the training-pushing expansion card according to any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the control method of the training and push expansion card according to any one of claims 1 to 8 are implemented.

Images