Control method of training expansion card, training expansion card and storage medium

Abstract

The application discloses a training and prediction expansion card control method, a training and prediction expansion card and a storage medium, and relates to the technical field of electronic digital data processing. The training and prediction expansion card control method comprises the following steps: if training and prediction unit output training and prediction data is detected, determining a video memory occupation value of the training and prediction data; if the video memory occupation value is greater than a video memory threshold value associated with the training and prediction unit, determining a target training and prediction expansion card; and storing the training and prediction data to the target training and prediction expansion card through a direct connection bus between the training and prediction unit and the target training and prediction expansion card. The technical problem that a large model cannot be trained and deployed locally due to hardware limitations of a graphics card is solved. A large model is trained and deployed locally at low cost, and a common computer is changed into a machine supporting large model training and prediction by inserting the training and prediction expansion card.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, and in particular to a control method for a training and pushing expansion card, the training and pushing expansion card, and a storage medium. Background Technology

[0002] Given the growing demand for data privacy protection, local deployment of large language models has become an important option for users to mitigate the risks of data leakage in the cloud. However, home computers typically have graphics cards with only 4GB-16GB of video memory, far below the minimum video memory requirements for running large models. This severe imbalance between video memory and the number of model parameters prevents users from training and deploying large models locally. Summary of the Invention

[0003] The main purpose of this application is to provide a control method for a training and push expansion card, a training and push expansion card, and a storage medium, which aims to solve the technical problem that the limitations of graphics card hardware prevent the local training and deployment of large models.

[0004] To achieve the above objectives, this application provides a control method for a training and push expansion card, the control method comprising:

[0005] If training data is detected to be output by the training unit, the video memory usage value of the training data is determined.

[0006] If the video memory usage value is greater than the video memory threshold associated with the training and push unit, the target training and push expansion card is determined;

[0007] The training and push data is stored in the target training and push expansion card via a direct connection bus between the training and push unit and the target training and push expansion card.

[0008] In one embodiment, before the step of determining the video memory usage value of the training data if training data is detected being output by the training unit, the method includes:

[0009] If at least one training and push expansion card is detected to be connected, load the driver configuration of the training and push expansion card;

[0010] Based on the driver configuration, the model of the training and push unit, and the hardware resources of the training and push expansion card access terminal, a large model training and push environment is configured on the terminal.

[0011] In one embodiment, after the step of configuring the large model training and push environment on the terminal, the following steps are included:

[0012] In response to the model training and inference command, determine the model identifier corresponding to the model training and inference command;

[0013] The data scheduling strategy corresponding to the model identifier is determined based on the large model training and push environment.

[0014] In one embodiment, after determining the resource scheduling strategy based on the model identifier and the configured large model training and push environment, the step includes:

[0015] Obtain model parameters;

[0016] Based on the data scheduling strategy, the model parameters are divided into video memory data and expansion card data;

[0017] The expansion card data is stored in the training expansion card, and the video memory data is sent to the training unit.

[0018] In one embodiment, before the step of dividing the model parameters into video memory data and expansion card data based on the data scheduling strategy, the following steps are included:

[0019] In response to a memory configuration command, determine the memory quota of the expansion card corresponding to the memory configuration command;

[0020] The data scheduling policy is updated according to the video memory quota.

[0021] In one embodiment, the training and push expansion card includes a processing core and storage chips. The processing core is programmed with a Linux system. The control method for the training and push expansion card further includes:

[0022] In response to the master card configuration command, determine the first training extension card corresponding to the master card configuration command;

[0023] The first training and push expansion card runs the Linux system, while the second training and push expansion card's Linux system is disabled, wherein the first training and push expansion card and the second training and push expansion card form an expansion card array.

[0024] In one embodiment, the control method for the training expansion card further includes:

[0025] In response to a pre-caching instruction, the target training extension card is determined according to the pre-caching instruction;

[0026] Determine the prefetched data in the target training extension card;

[0027] The prefetched data is transmitted to the training and push unit via a direct connection bus.

[0028] In one embodiment, before the step of determining the video memory usage value of the training data, the method further includes:

[0029] Determine whether the training data is used for calculation in the next frame;

[0030] If it is not used for calculation in the next frame, perform the step of determining the video memory usage value of the training data.

[0031] In addition, to achieve the above objectives, this application also provides a training and push expansion card, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the control method for the training and push expansion card as described above.

[0032] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, and stores a program implementing a control method for a training and pushing expansion card. The program implementing the control method for the training and pushing expansion card is executed by a processor to implement the steps of the control method for the training and pushing expansion card as described above.

[0033] This application provides a control method for a training and push expansion card. First, if training and push data is detected being output by the training and push unit, the memory usage value of the training and push data is determined. If the memory usage value is greater than the memory threshold associated with the training and push unit, a target training and push expansion card is identified. The training and push data is then stored on the target training and push expansion card via a direct connection bus between the training and push unit and the target training and push expansion card. This solves the technical problem of being unable to train and deploy large models locally due to graphics card hardware limitations; and enables low-cost local deployment of large models. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A flowchart illustrating the control method for the training and expansion card in this application (Example 1).

[0037] Figure 2 A flowchart illustrating the second embodiment of the control method for the training and extension card in this application;

[0038] Figure 3 A flowchart illustrating the control method for the training and expansion card in Embodiment 3 of this application;

[0039] Figure 4 This is a schematic diagram of a hardware structure of the training and expansion card of this application;

[0040] Figure 5 This is a schematic diagram of another hardware structure of the expansion card in this application.

[0041] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0042] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0043] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0044] Currently, with the increasing demand for data privacy protection, local deployment of large language models has become an important option for users to avoid the risk of data leakage in the cloud. However, the graphics cards of individual users' home terminals are usually equipped with 4GB-16GB of video memory, far below the minimum video memory requirements for running large models. This severe imbalance between video memory and the number of model parameters makes it impossible for users to train and deploy large models locally.

[0045] The main solution of this application is: if training push unit outputs training push data, determine the video memory usage value of the training push data; if the video memory usage value is greater than the video memory threshold associated with the training push unit, determine the target training push expansion card; and store the training push data to the target training push expansion card through the direct connection bus between the training push unit and the target training push expansion card.

[0046] This application achieves low-cost local deployment of a large model by connecting the training and push expansion card to the terminal, then interacting with the terminal's training and push unit via a direct-connect bus, and combining dynamic scheduling of training and push data.

[0047] It should be noted that the execution subject in this embodiment can be a training and push expansion card, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, personal computer, mobile phone, etc. This embodiment does not specifically limit it.

[0048] Based on this, Embodiment 1 of this application proposes a control method for a training and expansion card. Please refer to... Figure 1 The control method for the training expansion card includes steps S10 to S30:

[0049] Step S10: If training data is detected to be output by the training unit, determine the memory usage value of the training data.

[0050] It should be noted that the training and push expansion card in this application includes a processing core and storage chips. The processing core is programmed with a Linux system, which can directly execute the control methods of the training and push expansion card in this application to complete interaction with the training and push unit. Alternatively, the processor of the terminal connected to the training and push expansion card can execute the control methods of the training and push expansion card in this application to complete interaction between the training and push expansion card and the training and push unit. Furthermore, the terminal's processor can be the main system, and the Linux system of the training and push expansion card can be the secondary system to complete the interaction. Specifically, when the two systems coexist, the main system is used to access data from the training and push expansion card, and the secondary system is used to schedule data between the training and push expansion card arrays.

[0051] In this embodiment, the training and inference unit is the device that undertakes the main computational tasks during deep learning training and inference. It is typically a high-performance GPU (Graphics Processing Unit), such as NVIDIA's A100 or V100 graphics cards. These devices possess powerful parallel computing capabilities, enabling them to quickly process large-scale matrix operations and are the core hardware for deep learning model training and inference. Training and inference data refers to the data generated during the model training and inference phases, including input training samples, intermediate activation values ​​of the model, gradient data, model parameters, optimizer states, checkpoints, etc. For example, in image recognition model training, training and inference data can include image data, feature maps output from convolutional layers, and gradients obtained through backpropagation. Memory usage refers to the amount of storage space occupied by the training and inference data in the training and inference unit's memory, measured in bytes. Memory is the high-speed memory used by the GPU for temporary data storage; the memory usage reflects the consumption of memory resources by the training and inference data.

[0052] As an optional implementation, a data monitoring mechanism can be set at the output port of the training and push unit. This data detection function can be implemented through hardware circuitry or software drivers. For example, a flag bit can be set in the GPU's output register. When new training and push data is generated, the flag bit is set, thereby triggering the data detection program. After detecting the output of training and push data, the data type of the training and push data must first be identified. Common data types include integers, floating-point numbers, and Boolean values. Different data types occupy different storage spaces in video memory. For example, a 32-bit floating-point number typically occupies 4 bytes of memory space. For multidimensional data (such as tensors), it is necessary to calculate the size of each dimension. By multiplying the number of bytes occupied by the data type by the total number of elements in the data (i.e., the product of the sizes of each dimension), the video memory usage value of the training and push data can be obtained. For example, a 32-bit floating-point tensor with the shape (3,4,5) has a total of 3×4×5=60 elements, and each element occupies 4 bytes. Therefore, the video memory usage value of this tensor is 60×4=240 bytes.

[0053] Step S20: If the video memory usage value is greater than the video memory threshold associated with the training and push unit, determine the target training and push expansion card.

[0054] In this embodiment, the video memory threshold is an upper limit set for the video memory usage of training and inference data. When the video memory usage of training and inference data exceeds this threshold, it means that the video memory usage of the training and inference unit is close to or has reached saturation. At this time, it is necessary to transfer some training and inference data to other storage devices to avoid video memory overflow that could lead to training or inference interruption.

[0055] Target training and push expansion card: A specific expansion card selected from multiple training and push expansion cards, used to store training and push data. The training and push expansion card is connected to the training and push unit through a high-speed interface (such as PCIe), has a large storage capacity, and can alleviate the memory pressure on the training and push unit.

[0056] As an optional implementation, the calculated memory usage of the training data is compared with a memory threshold associated with the training unit. The memory threshold can be set based on the memory capacity of the training unit and actual application requirements. For example, for a GPU with 8GB of memory, the memory threshold can be set to 6GB. When the memory usage exceeds the memory threshold, relevant information from all training expansion cards is collected, including the remaining storage capacity, read / write speed, and health status of the expansion cards. This information can be obtained by querying the expansion card's management register or calling the expansion card's driver interface. Based on the collected expansion card information, a selection strategy is used to determine the target training expansion card. Common selection strategies include selecting the expansion card with the largest remaining storage capacity or the expansion card with the fastest read / write speed. For example, selecting the expansion card with the largest remaining storage capacity as the target expansion card ensures sufficient space to store the training data.

[0057] As another optional implementation, the usage status of each training and push expansion card is obtained, a preset data scheduling strategy is obtained, and the target training and push expansion card is determined based on the data scheduling strategy and the usage status of each training and push expansion card.

[0058] Specifically, a data scheduling strategy is a pre-defined set of rules and methods used to rationally allocate and schedule training and push data among multiple training and push expansion cards. Its purpose is to fully utilize the performance and resources of each training and push expansion card, improve the efficiency and performance of the entire training and push system, and avoid situations where some expansion cards are overused while others are idle. The data scheduling strategy can comprehensively consider multiple factors of the training and push expansion cards, such as remaining storage capacity, read / write speed, computing power, and load conditions, to achieve optimal data allocation and processing.

[0059] The usage of the training and push expansion card involves several key metrics, which can be obtained as follows: Remaining Storage Capacity: This is achieved by communicating with the training and push expansion card to query its storage device status information. For example, for flash-based training and push expansion cards, specific storage management protocols or driver interfaces can be used to obtain the remaining available storage space. In Linux systems, storage information can be read by calling relevant system commands or writing programs. Read / Write Speed: The read / write speed can be calculated by writing and reading data blocks of a certain size to the training and push expansion card, recording the time required for each operation, and then calculating the read / write speed. For example, using standard file read / write testing tools, a test file of a specific size can be created on the expansion card, and the time for writing and reading the file can be recorded to obtain numerical values ​​for the read / write speed. Computational Power: The computational power of the training and push expansion card can be evaluated by running benchmark programs. For example, benchmark tools provided by deep learning frameworks can be used to test the expansion card with common deep learning computational tasks such as matrix operations and convolution operations, and the computational power can be evaluated based on the test results. Load Status: The load status can be determined by real-time monitoring of the number of tasks being processed by the training and push expansion card, the complexity of the tasks, and the processing progress. This information can be obtained through the expansion card's monitoring interface or system logs.

[0060] Data scheduling policies are typically set during system initialization and stored in system configuration files or a database. The storage location of the data scheduling policy must be clearly defined; it could be a local configuration file (e.g., JSON or XML format) or a remote database. Depending on the storage location, appropriate methods are used to retrieve the data scheduling policy. If it's a local configuration file, file reading functions can be used to read the file content; if it's a remote database, database connection tools and query statements can be used to obtain the policy information.

[0061] Optionally, the training and push expansion card with the largest remaining storage capacity is selected as the target expansion card to ensure sufficient space for storing training and push data. The remaining storage capacities of each training and push expansion card are sorted. The training and push expansion card with the largest remaining storage capacity is selected as the target expansion card.

[0062] Optionally, the training and push expansion card with the fastest read and write speed is selected as the target expansion card to improve data transmission and processing efficiency. The read and write speeds of the acquired training and push expansion cards are sorted. The training and push expansion card with the fastest read and write speed is selected as the target expansion card.

[0063] Optionally, considering multiple factors such as the remaining storage capacity, read / write speed, computing power, and load of the training and push expansion card, a comprehensive score is calculated for each expansion card, and the expansion card with the highest score is selected as the target expansion card. A weight is assigned to each evaluation factor; for example, the weight of remaining storage capacity is 0.3, read / write speed is 0.3, computing power is 0.2, and load is 0.2. The evaluation factors for each training and push expansion card are standardized so that their values ​​range from 0 to 1. Based on the weights and standardized evaluation factor values, the comprehensive score for each training and push expansion card is calculated. The calculation formula is: Comprehensive Score = Remaining Storage Capacity Score × 0.3 + Read / Write Speed ​​Score × 0.3 + Computing Power Score × 0.2 + Load Score × 0.2. The training and push expansion card with the highest comprehensive score is selected as the target expansion card.

[0064] Step S30: The training and push data is stored in the target training and push expansion card through the direct connection bus between the training and push unit and the target training and push expansion card.

[0065] In this embodiment, the direct-connect bus is a high-speed data transmission channel, such as a PCIe bus, that directly connects the training / pushing unit and the training / pushing expansion card. The direct-connect bus provides low-latency, high-bandwidth data transmission capabilities, enabling rapid transmission of training / pushing data from the training / pushing unit to the target training / pushing expansion card. The training / pushing expansion card connects to the PCIe bus via PCIe P2P, allowing direct data interaction between the training / pushing expansion card and the training / pushing unit.

[0066] Furthermore, the training and push expansion cards interact via CXL (Compute Express Link), thereby achieving storage pooling. That is, each training and push expansion card selects a portion of its capacity to form a shared storage unit. The shared storage unit interacts with the training and push unit via the PCIe bus.

[0067] As an optional implementation, the training and push expansion card includes a cache area and a storage area, with the read / write speed of the cache area being greater than that of the storage area. Each training and push expansion card interacts with its cache area via CXL to pool the storage, so that if the cache area of ​​the target training and push expansion card is insufficient during training and push data writing, the training and push data is transferred in batches to the shared storage unit via CXL. Then, the background process transfers the training and push data from the cache area to the storage area, thereby improving the write speed of the training and push data.

[0068] Furthermore, if the training and push unit supports CXL, then each training and push expansion card is directly connected to the training and push unit via CXL, or each training and push expansion card is pooled via CXL storage, and then the common storage unit is directly connected to the training and push unit via CXL.

[0069] As an optional implementation, some preparatory work is required before data transmission, such as configuring the transmission parameters of the direct-connect bus (e.g., transmission rate, data block size) and initializing the data transmission buffer. This configuration can be accomplished by setting the registers of the bus controller. The data transmission process is then initiated, transferring training data from the training unit's video memory to the target training expansion card's storage medium via the direct-connect bus. During transmission, DMA (Direct Memory Access) technology can be used, allowing data to be transferred directly between the training unit and the target training expansion card without CPU intervention, thereby improving data transmission efficiency. After data transmission is complete, the target training expansion card sends a transmission confirmation message to the training unit, indicating that the data has been successfully stored. Upon receiving the confirmation message, the training unit releases the relevant video memory resources.

[0070] For example, a language model based on the Transformer architecture is trained using the TensorFlow framework. The training and induction unit is an NVIDIA RTX 4090 GPU with 24GB of VRAM and a VRAM threshold of 100MB. Four expansion cards (X, Y, Z, and W) are connected to the system, with remaining storage capacities of 129GB, 73GB, 529GB, and 326GB, respectively. During training, the training and induction unit (RTX 4090 GPU) performs forward and backward propagation calculations, generating intermediate activation values ​​and gradient data as training and induction data. When gradient data output is detected, its data type is identified as a 16-bit floating-point number, and its dimension is calculated. Assuming the gradient tensor has a shape of (32, 64, 128, 256), its total number of elements is 32 × 64 × 128 × 256 = 67,108,864. Each element occupies 2 bytes, so the video memory usage of this gradient data is 67,108,864 × 2 = 134,217,728 bytes (approximately 128 MB). This video memory usage will exceed the 100 MB threshold. Therefore, the gradient data needs to be stored in the training expansion card. Information on the remaining storage capacity of the four training expansion cards was collected, and expansion card Z was found to have the largest remaining storage capacity, at 529 GB. Therefore, expansion card Z was selected as the target training expansion card. The PCIe bus transmission parameters were configured, setting the transmission rate to the highest level, and the data transfer buffer was initialized. DMA technology was used to quickly transfer the gradient data from the training unit's video memory to the storage medium of expansion card Z via the PCIe bus.

[0071] As another optional implementation, if the video memory usage value is greater than the video memory threshold associated with the training and push unit, a target training and push expansion card is determined, wherein the target training and push expansion card is all training and push expansion cards. The number of training and push expansion cards is determined, the training and push data is divided into a number of parts corresponding to the number of training and push expansion cards, and each part of the training and push data is stored in the corresponding training and push expansion card.

[0072] For example, the system connects six training and push expansion cards: expansion card 1, expansion card 2, expansion card 3, expansion card 4, expansion card 5, and expansion card 6, with remaining storage capacities of 178GB, 73GB, 529GB, 326GB, 138GB, and 69GB, respectively. During training, the training and push unit performs forward and backward propagation calculations to generate intermediate activation values ​​and gradient data as training and push data. The GPU memory usage of the training and push data is 4.8GB. At this point, six training and push expansion cards are selected, and the training and push data is divided into six parts. The division method can be equal division or proportional allocation based on the remaining storage capacity of each training and push expansion card. After determining the six training and push sub-data parts, each training and push sub-data part is stored in the corresponding training and push expansion card.

[0073] At this point, by assembling the training and push expansion cards into RAID0, and then dividing the training and push data into n parts, where n is the number of training and push expansion cards, the technical effect of increasing the training and push data transmission speed to n times the original speed is achieved.

[0074] This embodiment selects a suitable target training and push expansion card based on the memory usage and memory threshold of the training and push data, and stores the training and push data in the expansion card through a direct connection bus, thereby effectively reducing the memory pressure on the training and push unit and ensuring the smooth progress of model training.

[0075] Based on the above embodiments, Embodiment 2 of this application proposes a control method for a training and pushing expansion card, referring to... Figure 2 Before step S10, the following are included:

[0076] Step A10: If at least one training and push expansion card is detected to be connected, load the driver configuration of the training and push expansion card.

[0077] In this embodiment, the driver configuration is a set of software programs and settings information used to enable the terminal operating system to recognize, control, and communicate with the expansion card. The driver configuration includes information such as the expansion card's hardware parameters, functional characteristics, and interfaces for interaction with the operating system, ensuring the expansion card functions properly.

[0078] The terminal's hardware management module continuously monitors the status of specific interfaces (such as PCIe slots). When a training expansion card is inserted into the interface, the hardware management module detects changes in interface levels and signals, thus confirming the connection of the training expansion card. For example, the PCIe interface detects the electrical connection of a new device, triggering the system's device detection mechanism. The terminal operating system uses the detected hardware identification information of the training expansion card (such as vendor ID and device ID) to search for a matching driver configuration file in the local driver library or a remote driver server connected to the network. The driver library is typically stored in a specific directory of the operating system and contains drivers for various hardware devices. Once a suitable driver configuration file is found, the operating system loads it into memory and performs initialization. The initialization process includes allocating system resources, setting hardware registers, and establishing a communication channel with the expansion card, enabling the training expansion card to interact normally with the terminal system.

[0079] Step A20: Based on the driver configuration, the model of the training and push unit, and the hardware resources of the training and push expansion card access terminal, configure a large model training and push environment on the terminal.

[0080] In this embodiment, different training and inference units vary in computing performance, memory capacity, and power consumption, which affect the efficiency and effectiveness of large-scale model training and inference. Hardware resources: These refer to the various hardware components and their performance indicators possessed by the terminal, including the number of CPU (Central Processing Unit) cores and clock speed, memory capacity, and read / write speed of storage devices. The availability of hardware resources limits the scale and speed of large-scale model training and inference, and must be fully considered when configuring the training and inference environment. Large-scale model training and inference environment: This is a comprehensive environment encompassing software and hardware settings, used to support the training and inference of large-scale deep learning models. It includes the installation and configuration of the operating system, deep learning frameworks (such as TensorFlow and PyTorch), CUDA and other computing libraries, as well as the allocation and scheduling of hardware resources.

[0081] Integrate the loaded training and push extension card driver configuration with the terminal system to ensure the operating system can correctly recognize and call the extension card's functions. For example, in a Linux system, modify kernel modules and device driver parameters to enable the system to manage the extension card as a usable hardware device. Install compatible deep learning frameworks and computing libraries based on the training and push unit model. Different GPU models may require different versions of CUDA drivers and deep learning frameworks to achieve optimal performance. For example, newer NVIDIA GPUs may require higher versions of CUDA to support their new computational features. Simultaneously, configure the framework to utilize the computing resources of the training and push unit and the training and push extension card. Conduct a comprehensive evaluation of the hardware resources connected to the training and push extension card on the terminal, including CPU performance, memory capacity, and storage speed. Based on the evaluation results, allocate hardware resources reasonably for large model training and push. For example, if memory capacity is limited, memory optimization strategies such as batch data loading and model parameter compression can be adopted. Set various environmental parameters required for large model training and push on the terminal, such as learning rate, batch size, and number of training epochs. These parameters affect the model's training effect and convergence speed and need to be adjusted according to the specific model and task. At the same time, configure data storage paths, log recording and other related settings to facilitate monitoring and management of the training and push process.

[0082] For example, consider training and inference of a large language model with 500 bytes of parameters on a personal PC. The PC originally had an NVIDIA RTX 3080 GPU as the training and inference unit. The user purchased two new training and inference expansion cards, which are connected to the PC via the PCIe interface. When the user inserts the two expansion cards into the PCIe slots, the PC's hardware management module detects the interface status change and confirms the connection of the expansion cards. The operating system, based on the expansion card's hardware identification information, finds the matching driver configuration file in the local driver library and loads it into memory for initialization. After initialization, the system successfully recognizes and can communicate with the expansion cards. The operating system integrates the expansion card's driver configuration with the system, and the newly connected expansion card device can be seen in the system's device manager. Since the training and inference unit is an NVIDIA RTX 3080 GPU, the user installed a compatible CUDA 11.3 driver and the PyTorch 1.10 deep learning framework. PyTorch is configured to utilize the computing resources of both the RTX 3080 and the expansion cards simultaneously. The PC's hardware resources were evaluated: an Intel Core i9-12900K CPU with 16 cores and 24 threads, 64GB of RAM, and a 1TB NVMe SSD for storage. Based on these resources, a data batch loading strategy was adopted to avoid memory overflow. In PyTorch, the training and induction environment parameters for the large language model were configured: a learning rate of 0.0001, a batch size of 16, and 10 training epochs. Simultaneously, a specific directory on the SSD was configured as the data storage path for fast data read and write, and a log file was set up to monitor metrics such as loss and accuracy during the training and induction process.

[0083] By following the steps above, the large model training and push environment was successfully configured on the terminal.

[0084] Optionally, after step A20, the following steps are included:

[0085] Step A30: In response to the model training and induction command, determine the model identifier corresponding to the model training and induction command.

[0086] In this embodiment, the model training / inference command is a command issued by the user or system to trigger the training or inference operation of a large model. It contains basic information about the training / inference task, such as the model to be used and the specific requirements for training / inference. The model identifier is a unique identifier assigned to each large model to distinguish different large models. The model identifier allows for accurate location and invocation of the corresponding large model for training / inference operations.

[0087] As an optional implementation, the terminal system monitors the input of model training and induction commands in real time. Upon receiving a command, it parses it and extracts key information. Model training and induction commands can be input in various ways, such as command-line input or graphical interface operation. The parsing process converts the command string into a data structure that the system can understand. In the parsed command data structure, fields related to the model identifier are searched. This field is usually predefined and used to explicitly specify the large model to be used. For example, the command might contain information like "model_id:123", where "123" is the corresponding model identifier.

[0088] Step A40: Determine the data scheduling strategy corresponding to the model identifier based on the large model training and push environment.

[0089] In this embodiment, the data scheduling strategy is a pre-defined set of rules and methods for the rational allocation and scheduling of training and push data for a specific large model. It takes into account various factors in the training and push environment, such as hardware performance and data characteristics, to optimize the efficiency and performance of the training and push process.

[0090] As an optional implementation, a comprehensive review of the previously configured large-scale model training and push environment is conducted, collecting key information. This information includes the model and performance parameters of the training and push units, the number of training and push expansion cards, storage capacity and read / write speed, operating system type and version, and deep learning framework configuration. A mapping table between model identifiers and data scheduling strategies is established; this table can be stored in the system configuration file or a database. Based on the determined model identifier, the corresponding data scheduling strategy is looked up in the mapping table. For example, if the model identifier is "123", the mapping table may record the "comprehensive performance evaluation strategy" corresponding to that model. For instance, in the comprehensive performance evaluation strategy, different training and push environments may need to adjust the weights of various evaluation factors such as remaining storage capacity and read / write speed to ensure that the strategy better adapts to the actual situation.

[0091] Through the above steps, the system determines the model identifier based on the model training and induction instructions, and determines a suitable data scheduling strategy for the model based on the large model training and induction environment, so as to improve the efficiency of local deployment and training of large models.

[0092] Optionally, after step A40, the following steps are included:

[0093] Step A50: Obtain model parameters.

[0094] In this embodiment, model parameters are a set of numerical values ​​learned by the large model during training, which determine the model's behavior and performance. For example, in a neural network model, model parameters include connection weights between neurons, bias terms, etc. They are the core components of the model, used to perform calculations and predictions on the input data.

[0095] As an optional implementation, model parameters may be stored in a specific folder on the local disk, or on a remote server or cloud storage. The storage location can be specified through configuration files, environment variables, or hard-coded methods. Depending on the determined storage location, the appropriate read method is used to retrieve the model parameters. For local storage, file read functions can be used; for remote storage, network requests and corresponding protocols (such as HTTP or FTP) are required to download the parameters. For example, in Python, the `torch.load()` function is used to read parameters from a PyTorch model.

[0096] Step A60: Based on the data scheduling strategy, the model parameters are divided into video memory data and expansion card data.

[0097] In this embodiment, the video memory data refers to the portion of data allocated from the model parameters that is suitable for storage in the training and induction unit's video memory. Video memory features high-speed read / write capabilities; storing frequently used data in video memory can improve the speed of model training and induction. The expansion card data refers to the portion of data allocated from the model parameters that needs to be stored in the training and induction expansion card. When the model parameters are too large for the training and induction unit's video memory to fully accommodate, some data is stored in the training and induction expansion card to alleviate video memory pressure.

[0098] As an optional implementation, model parameters are evaluated based on a previously determined data scheduling strategy, such as a remaining storage capacity priority strategy, a read / write speed priority strategy, or a comprehensive performance evaluation strategy. For example, under a remaining storage capacity priority strategy, the remaining space of the training and push unit's video memory and the remaining storage capacity of the training and push expansion card are given priority. Based on the evaluation results, the model parameters are divided into video memory data and expansion card data. This division can be based on factors such as the frequency of parameter usage and importance. For example, frequently accessed parameters are used as video memory data, while less frequently used parameters are used as expansion card data. Alternatively, the parameters can be divided according to a certain ratio, such as using the first 10% of parameters as video memory data and the remaining 90% as expansion card data.

[0099] Step A70: Store the expansion card data in the training expansion card and send the video memory data to the training unit.

[0100] In this embodiment, it is ensured that the training and push expansion card and the terminal system are correctly connected and initialized, and a communication channel is established through the corresponding interface protocol (such as PCIe protocol). P2PDMA (Direct Memory Access) technology or other efficient data transmission methods are used to transfer data from the expansion card's memory to its storage medium. During the transmission process, the transmission status needs to be monitored to ensure that the data is stored accurately.

[0101] Furthermore, before sending the video memory data, check if the training and push unit has enough video memory to store this data. If the video memory is insufficient, release some unnecessary data or adjust the data segmentation strategy.

[0102] Optionally, the memory configuration in the data scheduling policy can be modified. Before or after determining the data scheduling policy corresponding to the model identifier based on the large model training and push environment, the user can operate the memory quota of each training and push expansion card in the BIOS or driver. In response to the memory configuration command, the memory quota of the training and push expansion card corresponding to the memory configuration command is determined; the data scheduling policy is updated according to the memory quota.

[0103] In this embodiment, the video memory configuration command is a command issued by the user or system to configure the video memory usage of the training expansion card. This command typically includes requirements for setting the video memory quota, and can be entered via command line, graphical interface, or system configuration file. Video memory quota: The upper limit of usable video memory allocated to the training expansion card. It specifies the maximum amount of video memory the training expansion card can use during training, in order to rationally manage and allocate the system's video memory resources.

[0104] As an optional implementation, the system monitors the input of memory configuration commands in real time. Upon receiving a command, it parses it to extract information related to the training expansion card and memory quota. For example, the command might be "set_memory_quota--card_id2--quota195GB", where "--card_id2" indicates that the training expansion card number to be configured is 2, and "--quota195GB" indicates that the memory quota to be allocated is 195GB. Based on the parsed training expansion card identifier (such as number, model, etc.), the corresponding training expansion card is identified in the system. This identification can be accomplished by querying the system's device list or using device management tools. The specific memory quota value is extracted from the command and used as the memory quota for that training expansion card. For example, 195GB in the above command is determined as the memory quota for training expansion card number 2.

[0105] In this embodiment, by detecting the access of the training and push expansion card, the driver configuration is automatically loaded to the terminal when the training and push expansion card is accessed. Then, based on the hardware resources of the terminal and the model of the training and push unit, the large model training and push environment is configured. Then, when the user deploys the large model, the system hardware resources are automatically scanned and a resource optimization matching scheme is formulated for the specific hardware configuration.

[0106] Based on any of the above embodiments, Embodiment 3 of this application proposes a control method for a training and pushing expansion card, referring to... Figure 3 It also includes:

[0107] Step B10: In response to the master card configuration command, determine the first training expansion card corresponding to the master card configuration command.

[0108] In this embodiment, the master card configuration command is a command issued by the user or system to designate a specific expansion card in the training and pushing expansion card array as the master card. This command can be input via command line, graphical interface, or system configuration file, and its purpose is to clarify which expansion card in the array will undertake the main control and coordination tasks. The first training and pushing expansion card is the training and pushing expansion card designated as the master card according to the master card configuration command in the training and pushing expansion card array. The master card will play a core role in the operation of the entire expansion card array, responsible for communicating with other expansion cards (the second training and pushing expansion card), coordinating data transmission, and task allocation. The training and pushing expansion card includes a processing core and storage chips; the processing core is programmed with a Linux system.

[0109] As an optional implementation, the system monitors the input of master card configuration commands in real time. Upon receiving a command, it parses it and extracts the identification information related to the training and push expansion card to be designated as the master card. For example, the command might be "set_master_card--card_id1", where "--card_id1" indicates that the training and push expansion card numbered 1 should be designated as the master card. Based on the parsed training and push expansion card identifier, the system searches for the corresponding training and push expansion card in the system's device management list. The system checks whether the expansion card exists in the expansion card array and whether its status is normal. The expansion card can be accurately identified by querying its hardware ID, serial number, and other information. The training and push expansion card that matches the command requirements is determined as the first training and push expansion card (master card).

[0110] Step B20: Run the Linux system of the first training and push expansion card and disable the Linux system of the second training and push expansion card, wherein the first training and push expansion card and the second training and push expansion card form an expansion card array.

[0111] In this embodiment, the Linux system is the operating system running within the processing core of the training and push expansion card, providing support for hardware resource management, task scheduling, and data processing of the expansion card. The Linux systems on different training and push expansion cards can run independently, but in the expansion card array, they need to be coordinated according to the configuration of the primary card. The second training and push expansion card refers to the other training and push expansion cards in the array besides the first training and push expansion card (primary card). These expansion cards will work under the coordination of the primary card, and the running status of their Linux systems will be adjusted according to the primary card configuration. The expansion card array is a collection of multiple training and push expansion cards connected and organized in a certain way, aiming to improve the overall performance and storage capacity of the training and push system through collaborative work.

[0112] The system sends a boot signal to the first training push expansion card, triggering the startup of the Linux system within its processing kernel. During the boot process, the Linux system performs a series of initialization operations, including loading drivers, allocating system resources, and starting system services. To avoid conflicts and interference between multiple Linux systems, the system takes measures to disable the Linux system on the second training push expansion card. This disabling can be achieved through hardware switches, software commands, or modifications to system configurations. For example, by sending specific control signals to the second training push expansion card, its Linux system can be put into hibernation or prevented from starting certain critical services. After completing the booting of the Linux system on the first training push expansion card and the disabling of the Linux system on the second training push expansion card, the system continuously monitors the status of each expansion card.

[0113] In this embodiment, by configuring and running the Linux system of the first training and push expansion card, the system latency is further reduced, so that the terminal's processor does not need to participate in training and push, and even if the terminal's storage medium is removed or damaged, training and push can still continue.

[0114] Based on the above embodiments, Embodiment 4 of this application proposes a control method for a training and pushing expansion card, which further includes:

[0115] Step C10: In response to the pre-caching instruction, determine the target training expansion card according to the pre-caching instruction.

[0116] In this embodiment, the pre-caching instruction is an instruction issued by the system or user to trigger a pre-caching operation. During the training and induction of a large model, in order to improve the efficiency of data processing, some data is cached from the training and induction expansion card to the training and induction unit in advance, and the pre-caching instruction is used to initiate this operation.

[0117] As an optional implementation, the system monitors the input of pre-caching commands in real time. Upon receiving a command, it parses it to extract key information related to the target training and push expansion card, such as the card's identifier (number, model, etc.). The command may be entered as a command line, such as "precache --card_id2", indicating that the training and push expansion card with the number 2 should be selected as the target card. Based on the parsed card identifier, the system searches for the corresponding card in its device list.

[0118] Step C20: Determine the prefetch data in the target training extension card.

[0119] In this embodiment, prefetching data refers to data selected from the target training and induction expansion card and prepared for early transmission to the training and induction unit. This data is typically used during the training and induction process, such as model parameters and training samples. Cacheing it in the training and induction unit in advance can reduce the data reading waiting time and improve training and induction efficiency.

[0120] As an optional implementation, the data to be prefetched from the target training and push expansion card is determined according to the data access rules specified in the pre-caching instructions. These rules can be formulated based on factors such as data usage frequency, access order, and data dependencies. For example, data used frequently can be prefetched first. Data that matches the prefetching rules is located in the storage granules of the target training and push expansion card. Data can be quickly located using indexes, tags, etc. Simultaneously, the located data is filtered to remove unnecessary parts, ensuring that only data relevant to the current training and push task is prefetched.

[0121] Step C30: The prefetched data is transmitted to the training and push unit via a direct connection bus.

[0122] In this embodiment, the prefetch data transmission process is initiated through the direct bus interface. DMA (Direct Memory Access) technology is employed to allow data to be transferred directly between the target training expansion card and the training unit, reducing CPU intervention and improving transmission speed.

[0123] For example, in a deep learning-based image recognition model training and inference system, the system receives a pre-caching instruction, requiring it to pre-fetch some model parameters from the training and inference expansion card to the training and inference unit to accelerate subsequent inference. The system has three training and inference expansion cards, numbered 1, 2, and 3. The system receives the pre-caching instruction "precache--card_id2". After parsing the instruction, it searches for the training and inference expansion card numbered 2 in the system device list, confirms its normal operation and good connection to the system, and identifies it as the target training and inference expansion card. The pre-caching instruction specifies that data is pre-fetched according to data usage frequency. In the storage granules of the target training and inference expansion card, the model parameter data with high usage frequency is located by index. This data is filtered to remove parts irrelevant to the current image recognition task, determining the data to be pre-fetched. The PCIe bus transfer rate is configured to 32GB / s, and the data block size is 1MB. DMA transfer is initiated, transferring the pre-fetched model parameter data from the target training and inference expansion card to the training and inference unit via the PCIe bus.

[0124] In this embodiment, model parameters are processed and allocated appropriately at the start of training, and subsequently, the required expansion card data is loaded into the training and inference unit's GPU memory according to pre-caching instructions, providing necessary data support for the training and inference unit's computation. Combined with subsequent training and inference data processing steps, the use of GPU memory for large model training and inference can be effectively optimized, reducing the demand for high-capacity GPU memory and improving overall training and inference efficiency.

[0125] Based on the above embodiments, Embodiment 5 of this application proposes a control method for a training and pushing expansion card, which further includes the following steps before step S10:

[0126] Step D1: Determine whether the training data is used for calculation in the next frame;

[0127] Step D2: If not used for calculation in the next frame, perform the step of determining the video memory usage value of the training data.

[0128] In this embodiment, the next frame calculation refers to the calculation steps performed sequentially in chronological order during the model's training or inference process. The next frame calculation refers to the calculation step immediately following the current one. Whether the training and inference data is used for the next frame calculation determines whether that data needs to be immediately retained in the training and inference unit's memory to meet the requirements of fast access.

[0129] As an alternative implementation, the computational graph of large models is extremely complex, containing thousands or even tens of thousands of nodes and edges. Advanced graph analysis algorithms are used to deeply analyze the computational graph, clarifying the dependencies and usage positions of each training and inference data point. For example, in a large language model based on the Transformer architecture, the output data of certain intermediate layers may only be used in specific multi-head attention mechanisms or feedforward network modules. By analyzing the computational graph, it can be accurately determined whether this data will be used in the next frame's computation. During the training and inference process of large models, the computational progress and status are tracked in real-time and accurately. Using timestamps and event logs in the distributed system, the generation and usage times of each training and inference data point are recorded. Combined with the current computational step, it can be accurately determined whether the data will be used in the next computation. For example, in the distributed training of large models, the intermediate gradient data generated by some nodes may only be used for local updates in the current iteration step and will not be used in the next frame's computation. When it is determined that the training and inference data will not be used in the next frame's computation, the following operations are performed to determine its memory usage: The data types in large models are diverse and complex, including not only common integers, floating-point numbers, and booleans, but also custom data types. A specialized data type identification algorithm is employed to accurately identify the data type of the training and inference data. For example, for quantized data in large models, the algorithm accurately identifies the quantization bit depth and data format to determine the storage space occupied by each element. Data in large models typically has extremely high dimensionality; for example, the dimensions of tensors may reach tens or even hundreds. An efficient dimensionality calculation algorithm is used to accurately calculate the size of each dimension of the multidimensional data. The memory usage of the training and inference data is obtained by multiplying the number of bytes occupied by the data type by the total number of elements (i.e., the product of the sizes of each dimension). For example, for a 16-bit floating-point tensor with the shape (1024, 1024, 1024, 64), the total number of elements is 1024 × 1024 × 1024 × 64, and each element occupies 2 bytes. Therefore, the memory usage of this tensor is 1024 × 1024 × 1024 × 64 × 2 bytes. When the memory usage of the training and push data exceeds the memory threshold, the following steps are performed to store the training and push data in the training and push expansion card: A unique and detailed identifier is added to the training and push data, containing rich information such as data type, generation time, model layer, and data source node. Simultaneously, an efficient data index structure is constructed to facilitate rapid management and retrieval in the training and push expansion card. For example, a hash table or B-tree data structure is used to store the mapping relationship between data identifiers and storage locations. The training and push data is transferred from the graphics memory of the training and push unit to the storage medium of the training and push expansion card via a high-speed, low-latency interface (such as PCIe 5.0 or a higher level interface). Advanced data compression algorithms (such as Zstd compression algorithm) are used to efficiently compress the training and push data, significantly reducing data transmission volume and time. Furthermore, multi-channel parallel transmission technology is utilized to further improve transmission efficiency.Due to the massive amount of training and push data generated by large models, the training and push expansion card may employ a distributed storage architecture. When storing training and push data, a distributed file system (such as Ceph, GlusterFS, etc.) is used for management, distributing the data across multiple storage nodes. Simultaneously, the storage location and related metadata (such as identifier, size, checksum, etc.) of each data block are recorded to ensure data reliability and accessibility.

[0130] Through the above embodiments, it is determined whether the training and push data is used for the calculation of the next frame, and the unused data is stored in the training and push expansion card according to the memory usage value and memory threshold. This effectively reduces the demand for high memory capacity for large model training and improves training efficiency and resource utilization.

[0131] Based on any of the above embodiments, Embodiment Six of this application proposes a control method for a training and pushing expansion card. The training and pushing data includes at least one of model gradients, optimizer parameters, and checkpoints. The step of storing the training and pushing data to the target training and pushing expansion card through a direct bus between the training and pushing unit and the target training and pushing expansion card includes at least one of the following: unloading the model gradients to the training and pushing expansion card; and updating the optimizer state based on the optimizer parameters.

[0132] In this embodiment, the training and induction data includes at least one of the following: model gradient, optimizer parameters, and checkpoints. Model gradient: In large model training, the model gradient is the partial derivative of the objective function with respect to the model parameters. It represents the rate of change of the objective function with respect to the current parameter values, reflecting which direction adjusting the model parameters can reduce the objective function value, and is an important basis for updating model parameters. Optimizer parameters: The optimizer is used to adjust model parameters to minimize the loss function. Optimizer parameters are the parameters used by the optimizer during the update process, such as learning rate and momentum. Different optimizers have different parameter settings, which affect the training speed and effect of the model. Checkpoint: A checkpoint is a snapshot of the model during training, containing the model's state at a specific moment, such as the model's parameter values ​​and the optimizer's state. Saving checkpoints facilitates resuming training after an interruption and can also be used for model evaluation and comparison. Unloading: The process of moving data from the GPU memory of the training and induction unit to the training and induction expansion card. The purpose is to free up GPU memory space in the training and induction unit to cope with insufficient GPU memory. Optimizer state: The internal state of the optimizer during training. For example, when using a stochastic gradient descent (SGD) optimizer, the accumulated momentum is part of the optimizer state. The optimizer state is continuously updated as training progresses to guide the updating of model parameters.

[0133] As an optional implementation, when the training and induction data output by the training and induction unit contains model gradients, and the memory usage of the calculated model gradients exceeds the memory threshold associated with the training and induction unit, the following operations are performed: Data Tagging: A unique identifier is added to the model gradient data, containing information such as data type (model gradient), generation time, and model layer, facilitating subsequent management and retrieval in the training and induction expansion card. Data Transfer: The model gradient data is transferred from the memory of the training and induction unit to the storage medium of the training and induction expansion card via a high-speed interface (such as a PCIe interface). During the transfer, a data compression algorithm (such as Huffman coding) is used to compress the model gradient data to reduce the amount of data transferred and the transfer time. Storage Management: Storage space is allocated for the model gradient data in the training and induction expansion card, and its storage location and related metadata (such as identifier, size, etc.) are recorded. Storage management can be performed using a file system or database to quickly locate and access the model gradient data. As an optional implementation, when the training and induction data output by the training and induction unit contains optimizer parameters, the following operations are performed: Parameter Parsing: The optimizer parameters are parsed to extract the values ​​of each parameter, such as learning rate and momentum. Different optimizers have different parameter formats, requiring parsing based on the specific optimizer type. State Update: The internal state of the optimizer is updated based on the parsed optimizer parameters. For example, when using a stochastic gradient descent optimizer with momentum, the accumulated momentum value is updated based on the new momentum parameters. The update process must follow the optimizer's update rules to ensure the correctness of the optimizer state. State Saving: The updated optimizer state is saved to the GPU memory of the training unit or to the training expansion card. If GPU memory is sufficient, the optimizer state can be saved to GPU memory for fast access; if GPU memory is limited, the optimizer state is saved to the training expansion card, and its storage location is recorded.

[0134] Optionally, after the step of unloading the model gradient to the training and pushing extension card, the method includes: if a gradient update instruction is received from the training and pushing unit, obtaining the target model gradient corresponding to the gradient update instruction; and sending the target model gradient to the training and pushing unit.

[0135] In this embodiment, the gradient update instruction is a command issued by the training and induction unit to request specific model gradients for gradient update operations. This instruction includes relevant identifiers of the required model gradients, such as generation time and the model layer to which they belong, to accurately locate the target model gradient. The target model gradient is the model gradient data that the training and induction unit needs to use during the gradient update process, retrieved and determined from the training and induction extension card according to the gradient update instruction. Accumulated gradients: In some deep learning training strategies, to simulate larger batch sizes, model gradients are gradually accumulated over multiple training steps, and then used to update model parameters after a certain number of steps.

[0136] As an optional implementation, the training and push extension card listens for instructions from the training and push unit in real time. Upon receiving a gradient update instruction, it parses the instruction and extracts the identifier information of the target model gradient, such as generation time and model layer. Based on the parsed identifier information, it searches for the gradient in the training and push extension card's storage management system. When storing model gradient data, the training and push extension card records relevant metadata (such as identifier and storage location) for each data point, allowing for quick location of the target model gradient's storage location. After locating the target model gradient data, it verifies it to ensure data integrity and correctness. Verification can be performed using checksums, hash values, etc. If data verification fails, appropriate error handling measures are required, such as re-acquiring the data or reporting error information. The target model gradient data is read from the training and push extension card's storage medium. If the data was compressed during storage, such as using Huffman coding, decompression is required after reading the data. The read and decompressed target model gradient data is sent to the training and push unit's video memory via a high-speed interface.

[0137] Optionally, the training and push data includes model parameters. The step of storing the training and push data to the target training and push expansion card via the direct connection bus between the training and push unit and the target training and push expansion card includes: if the training and push data also includes checkpoints, associating the checkpoints with the model parameters and storing them to the training and push expansion card.

[0138] As an optional implementation, checkpoints and model parameters are stored in the training extension card in an interrelated manner, so that when needed later, the corresponding model parameters can be quickly located based on the checkpoints, or the corresponding checkpoint information can be found based on the model parameters.

[0139] Through the above embodiments, the gradient of the target model is obtained from the training-pull expansion card according to the gradient update instruction and sent to the training-pull unit to support the gradient accumulation training strategy, while effectively reducing the demand for high-capacity GPU memory for large model training. Model parameters that do not need to participate in the calculation are unloaded to the training-pull expansion card, and checkpoints are associated with model parameters for storage, thereby effectively reducing the demand for high-capacity GPU memory for large model training, while facilitating training recovery and model evaluation.

[0140] Based on the above embodiments, Embodiment 7 of this application proposes a control method for training and inference expansion cards, which further includes: if data to be inferred is received, obtaining the video memory remaining amount of each training and inference expansion card; if the video memory remaining amount is not lower than the running threshold, calling the video memory remaining amount to execute the inference process based on the data to be inferred, so as to output the inference result.

[0141] In this embodiment, the data to be inferred refers to the data that needs to be input into the large model for inference calculations in a real-time training (training while inference) scenario. For example, in a natural language processing task, the data to be inferred might be a piece of text; in an image recognition task, it might be an image. The remaining video memory of the training-inference expansion card refers to the amount of unused storage space in the current video memory of the training-inference expansion card. Since the training-inference expansion card is used to store some training-inference data to alleviate the video memory pressure on the training-inference unit, its remaining video memory reflects the available resources currently available for executing the inference process. The operating threshold is a minimum available space standard set for the video memory of the training-inference expansion card. When the remaining video memory is not lower than this threshold, it indicates that the training-inference expansion card has sufficient resources to execute the inference process; otherwise, it may not be able to perform inference normally. The inference process is the process by which the large model calculates based on the input data to be inferred using the learned model parameters, thereby deriving the inference result. In a real-time training scenario, the inference process and the training process run in parallel.

[0142] As an optional implementation, the training and inference unit interacts with the training and inference expansion card via a specific communication protocol (such as the PCIe interface communication protocol). The training and inference unit sends a request command to the training and inference expansion card to obtain the available video memory. After receiving the request command, the training and inference expansion card queries the video memory usage through its internal hardware management module. This module can directly access the video memory status register to obtain information on the used video memory space and the total video memory space. Based on the queried used video memory space and total video memory space, the available video memory of the training and inference expansion card is calculated. For example, if the total video memory of the training and inference expansion card is 1024GB and 20GB has been used, then the available video memory is 1004GB. When the available video memory of the training and inference expansion card is not lower than the operating threshold, the data to be inferred is loaded from the input device (such as a network interface, storage device, etc.) into the video memory of the training and inference expansion card. During the loading process, the data may need to be preprocessed, such as normalization, pruning, encoding, etc., to make it conform to the input requirements of a large model. The training and inference expansion card retrieves the relevant parameters of the large model from its storage and loads them into the video memory. Due to the massive number of parameters in the large model, techniques such as parameter partitioning and asynchronous loading may be necessary to improve loading efficiency. The training and inference expansion card invokes its computing cores (such as GPU cores or dedicated inference chips) to perform inference computation based on the loaded data to be inferred and the model parameters. During the computation, parallel computing techniques, such as multi-threaded computation and SIMD instruction sets, are used to accelerate the inference speed. After the inference computation is completed, the obtained inference results are output from the training and inference expansion card's video memory. The results can be returned to the requesting end (such as a client or application) or used for further processing and analysis.

[0143] Through the above embodiments, in the scenario of large-scale models with real-time training (training while inference), the decision to execute the inference process is based on the remaining memory of the training and inference expansion card. This fully leverages the large capacity of the training and inference expansion card to achieve efficient inference of large models, while reducing reliance on a large number of graphics cards, lowering costs and improving resource utilization.

[0144] This application provides a training and push expansion card, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the control method of the training and push expansion card in the first embodiment described above.

[0145] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a training expansion card suitable for implementing the embodiments of this application. Figure 4 The training expansion card shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0146] like Figure 4 As shown, the training expansion card may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the training expansion card. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the training expansion card to communicate wirelessly or wiredly with other devices to exchange data. Although training expansion cards with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0147] Reference Figure 5 , Figure 5As another example of the training and push expansion card in this embodiment, the training and push expansion card includes a processing core and storage particles. The processing core is programmed with a Linux system, and the storage particles are used to expand the video memory. After the training and push expansion card is connected to the terminal, it interacts directly with the terminal's training and push unit by running the Linux system to train and push large models and deploy large models locally.

[0148] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0149] The training and push expansion card provided in this application, employing the control method of the training and push expansion card in the above embodiments, can solve the technical problem that the limitation of graphics card hardware prevents the local training and deployment of large models. Compared with the prior art, the beneficial effects of the training and push expansion card provided in this application are the same as those of the training and push expansion card provided in the above embodiments, and other technical features in this training and push expansion card are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0150] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0151] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0152] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the control method of the training expansion card in the above embodiments.

[0153] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), or any suitable combination thereof.

[0154] The aforementioned computer-readable storage medium may be included in the training expansion card; or it may exist independently and not be assembled into the training expansion card.

[0155] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the training expansion card, cause the training expansion card to: determine the video memory occupancy value of the training data if it detects that the training unit outputs training data;

[0156] If the video memory usage value is greater than the video memory threshold associated with the training and push unit, the target training and push expansion card is determined;

[0157] The training and push data is stored in the target training and push expansion card via a direct connection bus between the training and push unit and the target training and push expansion card.

[0158] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0161] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the control method of the training and pushing expansion card described above. This solves the technical problem that limitations in graphics card hardware prevent the local training and deployment of large models. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the control method of the training and pushing expansion card provided in the above embodiments, and will not be elaborated upon here.

[0162] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described control method for a training expansion card.

[0163] The computer program product provided in this application can solve the technical problem that the limitations of graphics card hardware prevent the local training and deployment of large models. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the control method for training and pushing expansion cards provided in the above embodiments, and will not be repeated here.

[0164] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A control method for a training expansion card, characterized in that, The training and push expansion card includes a processing core and storage chips. The processing core is programmed with a Linux system. The control method for the training and push expansion card includes: If training data is detected to be output by the training unit, the video memory usage value of the training data is determined. If the video memory usage value is greater than the video memory threshold associated with the training and push unit, a comprehensive score is calculated for each training and push expansion card based on the remaining storage capacity, read and write speed, computing power and load conditions, and the training and push expansion card with the highest score is selected as the target training and push expansion card. The training and push data is stored in the buffer area of ​​the target training and push expansion card through the direct connection bus between the training and push unit and the target training and push expansion card. When the capacity of the buffer area is insufficient, the training and push data is transferred to a common storage unit. The common storage unit refers to a storage unit composed of the buffer areas of each training and push expansion card through a fast link interactive storage pool.

2. The control method for the training and expansion card as described in claim 1, characterized in that, Before the step of determining the video memory usage value of the training data if the training unit outputs training data, the following steps are included: If at least one training and push expansion card is detected to be connected, load the driver configuration of the training and push expansion card; Based on the driver configuration, the model of the training and push unit, and the hardware resources of the training and push expansion card access terminal, a large model training and push environment is configured on the terminal.

3. The control method for the training and expansion card as described in claim 2, characterized in that, Following the step of configuring the large model training and push environment on the terminal, the following steps are included: In response to the model training and inference command, determine the model identifier corresponding to the model training and inference command; The data scheduling strategy corresponding to the model identifier is determined based on the large model training and push environment.

4. The control method for the training and expansion card as described in claim 3, characterized in that, After determining the resource scheduling strategy based on the model identifier and the configured large model training and push environment, the following steps are included: Obtain model parameters; Based on the data scheduling strategy, the model parameters are divided into video memory data and expansion card data; The expansion card data is stored in the training expansion card, and the video memory data is sent to the training unit.

5. The control method for the training and expansion card as described in claim 4, characterized in that, Before the step of dividing the model parameters into video memory data and expansion card data based on the data scheduling strategy, the following steps are included: In response to a memory configuration command, determine the memory quota of the expansion card corresponding to the memory configuration command; The data scheduling policy is updated according to the video memory quota.

6. The control method for the training and expansion card as described in claim 1, characterized in that, The training and push expansion card includes a processing core and storage chips. The processing core is programmed with a Linux system. The control method for the training and push expansion card further includes: In response to the master card configuration command, determine the first training extension card corresponding to the master card configuration command; The first training and push expansion card runs the Linux system, while the second training and push expansion card's Linux system is disabled, wherein the first training and push expansion card and the second training and push expansion card form an expansion card array.

7. The control method for the training and expansion card as described in claim 1, characterized in that, The control method for the training and push expansion card also includes: In response to a pre-caching instruction, the target training extension card is determined according to the pre-caching instruction; Determine the prefetched data in the target training extension card; The prefetched data is transmitted to the training and push unit via a direct connection bus.

8. The control method for the training and expansion card as described in claim 1, characterized in that, Before the step of determining the video memory usage value of the training data, the method further includes: Determine whether the training data is used for calculation in the next frame; If it is not used for calculation in the next frame, perform the step of determining the video memory usage value of the training data.

9. A training expansion card, characterized in that, The training and push expansion card includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the control method for the training and push expansion card as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the steps of the control method for the training expansion card as described in any one of claims 1 to 8.

Images