A method for remote training of a large language model GPT-2 in a storage-compute separation scenario

Abstract

This invention provides a method for remote training of a large language model, GPT-2, in a compute-in-memory (CIDM) scenario, belonging to the fields of artificial intelligence and cloud computing technology. The method includes: establishing a communication connection between the client and server; the client serializing data, and the serialized data being transmitted to the server via the network; the server's main process creating a data receiving thread for data reception and feedback, with the received data stored in a shared queue; establishing a server-side multi-process distributed parallel training model, where each process retrieves data from the shared queue for model training, and the data receiving thread can receive data simultaneously during training, achieving parallel execution of training and receiving; and continuously performing data transmission and training tasks until a specified number of training rounds are completed or a termination condition is met. This invention achieves parallel execution of large language model training and data reception in a compute-in-memory (CIDM) scenario, improving training efficiency and enhancing model performance.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and cloud computing technologies, and in particular to a method for remote training of the large language model GPT-2 in a storage-compute separation scenario. Background Technology

[0002] In current AI model training, the demand for storage and computing resources is rising sharply as data volume and model size continue to increase. Traditional in-memory computing architectures bind computing and storage resources to a single node, resulting in low resource utilization, poor scalability, and inflexible resource allocation. For example, when computationally intensive tasks consume a large amount of CPU or GPU, storage resources may be idle, and vice versa. Adding hardware resources to nodes can only simultaneously improve computing and storage capabilities, and cannot expand a specific resource type independently, leading to high hardware upgrade costs and low efficiency.

[0003] In a compute-in-memory architecture, computing and storage resources cannot be dynamically scheduled according to the actual load, resulting in overall performance inefficiency. Furthermore, in distributed training scenarios, multiple nodes need to frequently access the same storage resources, increasing storage access latency and causing network bandwidth to become a performance bottleneck. In a compute-in-memory architecture, computing and storage resources are tightly coupled to a single node. Once a node fails, such as due to hardware damage or network connection interruption, not only will computing tasks stop immediately, but storage functionality will also be completely lost, resulting in inaccessible data.

[0004] The compute-in-memory (CIMM) architecture improves system resource utilization and scalability by decoupling compute and storage modules. With compute and storage nodes separated, CIMM allows for independent expansion of compute or storage resources based on actual needs, avoiding the waste caused by resource coupling in traditional architectures. CIMM allows for the independent addition of compute or storage nodes, enabling system expansion at a lower cost and supporting larger-scale deep learning training tasks. CIMM can deploy storage and compute resources across multiple geographical locations, making it particularly suitable for cloud-based deep learning tasks requiring distributed storage and computation. Because of the separation of storage and compute, storage nodes maintain data integrity even when compute nodes fail, preventing system crashes due to a single node failure.

[0005] The compute-in-memory (CIM) architecture, by decoupling compute and storage modules, provides more flexible resource scheduling and higher resource utilization. In traditional compute-in-memory architectures, compute and storage requirements are often out of sync. For example, highly compute-intensive tasks may require powerful computing capabilities, while data storage needs are relatively low. Data-intensive tasks require large-capacity storage resources, but compute resource utilization is low. This imbalance makes resource allocation difficult to optimize, leading to over-utilization of some resources and idle waste of others. Therefore, how to meet the different needs of storage and compute remains a pressing technical problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a remote training method for the large language model GPT-2 in a storage-computation separation scenario. This method decouples storage and computation, establishes a communication connection between the client and the server using TCP / IP, uses a desktop host as the client to store and send data, and remote computing resources as the server to undertake computation tasks. This solves the problem that the integrated storage-computation model cannot meet the different storage and computation needs.

[0007] To achieve the above objectives, this invention proposes a method for remote training of the large language model GPT-2 in a storage-computation separation scenario, comprising the following steps:

[0008] Step S1: The client establishes a communication connection with the server and sends data description information, which includes the training rounds, the number of training batches, and the number of test batches;

[0009] Step S2: The client serializes the data and sends it to the server via the network. The serialized data includes the first four bytes indicating the data length and the subsequent actual training data.

[0010] Step S3: The data receiving thread in the server's main process receives the serialized data, parses the first four bytes to determine the length of the data block, and stores the data in a shared queue;

[0011] Step S4: The server uses multiple processes to train the model. Each training process extracts data blocks from the shared queue, performs word segmentation, and then trains the GPT-2 model in parallel.

[0012] Step S5: Repeat steps S2 to S4 until the preset training termination condition is met;

[0013] Step S6: Evaluate the performance of the trained GPT-2 model.

[0014] Preferably, in step S1, before the client sends data to the server, it determines the client-server communication port, establishes a communication connection, selects the training batch size and training dataset. Specific steps include:

[0015] Step S11: Select a port number. When the client and server establish a communication connection, both ends need to select an available port for establishing the connection and transmitting data. You cannot select a system-reserved port number or a port number that has been occupied by an application.

[0016] Step S12: Establish a communication connection. The client uses the temporary port allocated by the operating system to establish a communication connection with the specified port of the server.

[0017] Step S13: Select the batch size based on the server-side computing resource hardware configuration.

[0018] Preferably, in step S2, after the client establishes a communication connection with the server, it needs to serialize data and send training round description information. Specific steps include:

[0019] Step S21: Serialize and send data, serialize the data into a byte stream and send the byte stream data;

[0020] Step S22: Send training round description information. Calculate the batch number information of the training dataset and the test dataset based on the number of dataset entries and batch size. Before sending the dataset, first send the number of training rounds and the number of batches contained in the training dataset and the test dataset in one training round.

[0021] Preferably, in step S3, the server-side data receiving thread performs the tasks of data reception, deserialization, and storage in the shared queue. The specific steps include:

[0022] Step S31: The server first parses the training data length information from the first four bytes of the received data, then receives the data in blocks and concatenates the data into a byte array;

[0023] Step S32: Deserialize the received byte stream data;

[0024] Step S33: Divide the received sequence data evenly according to the number of training processes, and store the divided data in a shared queue for use by the training processes.

[0025] Preferably, in step S4, the server also needs to establish a communication connection, select a model, create a data receiving thread, and run multiple training processes before training the model. Specific steps include:

[0026] Step S41: Establish a communication connection. The server listens for communication connection requests on a specified port and establishes a communication connection with the client.

[0027] Step S42: Construct a multi-process distributed parallel training model GPT-2 based on the generative pre-trained Transformer model. The GPT-2 model includes a word position embedding layer, a word embedding layer, 12 repeatedly stacked GPT blocks and a layer normalization layer. Each GPT block contains two layer normalization layers, an attention layer and a multilayer perceptron. The hidden layer dimension is 768.

[0028] Step S43: Create multiple training processes. The server-side model training uses a distributed training method, creating multiple processes to control multiple graphics card devices to execute the same training task. After the multi-process training is completed, the model parameters are updated simultaneously.

[0029] Step S44: Create a data receiving thread. Create a data receiving thread in the server's main process.

[0030] Preferably, in step S43, the server-side training process for training the model includes retrieving data from the shared queue, word segmentation, model training, backpropagation, gradient aggregation, and parameter updates. Specific steps include:

[0031] Step S431: Each training process retrieves data from the shared queue, determines whether there is data in the shared queue, and blocks and waits until there is data in the shared queue when the shared queue is empty.

[0032] Step S432: Use a tokenizer to convert continuous data into a series of sub-unit tokens;

[0033] Step S433: Train the model. Before training begins, clear the gradients of all parameters to zero. Use a sliding window mechanism to construct the source input and target output. The source input is mapped to a word embedding representation through the word embedding layer. The word embedding representation and the word position embedding representation are added element by element and then used as the model input. The model input is output after forward propagation. The model output is converted into a token representation after passing through the language head layer. Calculate the cross-entropy loss between the model output and the target output.

[0034] Step S434: Calculate the gradient through backpropagation, and temporarily store the calculated gradient result in the grad attribute of the model parameters;

[0035] Step S435: Aggregate gradients. Aggregate the gradient results obtained from multiple training processes using an average aggregation strategy, and distribute the aggregated results to each training process.

[0036] Step S436: Update the model parameters using the AdamW adaptive moment estimator with separated weight decay.

[0037] Preferably, to make the GPT-2 large language model more effective at processing time series data, the internal architecture of the GPT-2 model is improved, specifically as follows:

[0038] The spatiotemporal embedding layer adds base station ID embedding and multi-scale temporal embedding to the word embedding and location embedding built into the GPT-2 model;

[0039] The original attention layer of the GPT-2 model is improved into a dual-stream attention layer, which includes a temporal attention sub-layer and a spatial attention sub-layer, and is dynamically fused through a gating mechanism.

[0040] A graph convolutional layer is added after the dual-stream attention layer to capture base station topology information.

[0041] Preferably, in step S6, after the entire model training is completed, it is also necessary to evaluate the performance of the memory-compute separation architecture model. Specific steps include:

[0042] Step S61: For a single batch of data, record a timestamp at the end of each batch of training, and subtract adjacent timestamps to obtain the training time of a batch;

[0043] Step S62: Record the time for forward propagation, back propagation and gradient update for each batch of data as the computation time, and record the difference between the training time and computation time for a batch of data as the communication time;

[0044] Step S63: Record the training time, communication time, and computation time for each batch under the scenarios of storage-computation separation and storage-computation integration, and draw a comparison graph.

[0045] Therefore, this invention proposes a method for remote training of the large language model GPT-2 in a storage-computation separation scenario, the beneficial effects of which are as follows:

[0046] (1) The model remote training method in the scenario of storage-computation separation proposed in this invention breaks the binding of computing and storage on a single node, so that computing resources and storage resources can be flexibly and dynamically scheduled according to the actual load.

[0047] (2) By storing data on the client side and performing calculations on the server side, this invention can achieve efficient data transmission and calculation, and support tasks such as training large-scale deep learning models.

[0048] (3) The present invention uses an independent receiving thread and a shared queue to realize data receiving, deserialization and parallel model training, which partially overlaps the network communication time and the computing time, effectively masking the network delay and reducing the impact of communication overhead on the overall training speed.

[0049] (4) The general GPT model was improved by introducing spatiotemporal embedding, dual-stream attention and graph convolution layer, which enabled it to better capture the spatiotemporal dependencies in time series data and improve the performance of the model in tasks such as cellular traffic prediction. Attached Figure Description

[0050] Figure 1 A flowchart of a method for remote training of a large language model GPT-2 in a storage-compute separation scenario;

[0051] Figure 2 This is a schematic diagram of a method for remote training of a large language model GPT-2 in a storage-compute separation scenario.

[0052] Figure 3 A schematic diagram of the improved GPT-2 model architecture;

[0053] Figure 4 This is a comparison chart showing the training time for each batch in a storage-compute separation scenario with the training time for storage-compute integration. Detailed Implementation

[0054] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0056] Example

[0057] like Figure 1 and Figure 2 As shown, this invention provides a method for remote training of a large language model GPT-2 in a storage-computation separation scenario, comprising a client system and a server system. The client is responsible for storing and sending data, while the server undertakes the computation task. The specific implementation steps are as follows:

[0058] The specific implementation steps for the client in a storage-compute separation scenario include:

[0059] Step S1: The client meticulously divides the massive training data into independent data blocks and generates a unique index for each data block to enable rapid location and retrieval;

[0060] Step S2: The client establishes a communication connection with the server. Specific steps include:

[0061] Step S21: Select a suitable port number. When the client and server establish a communication connection, both ends need to select an available port for establishing the connection and transmitting data. Note that you cannot select a system-reserved port number or a port number that has been occupied by an application to ensure that the communication connection is established correctly.

[0062] Step S22: The client uses the temporary port allocated by the operating system to establish a communication connection with the specified port of the server;

[0063] Step S23: Select the batch size according to the server's computing resource hardware configuration. The batch size will affect the GPU memory usage during model training. Setting the batch size too large will lead to insufficient GPU memory.

[0064] Step S3: The client sends detailed data description information to the server, including the training rounds, the number of training data batches sent in each training round, and the number of test data batches sent.

[0065] In order for the server to know the batch number description information of the training dataset and the test dataset, the batch number information of the training dataset and the test dataset is first calculated based on the number of dataset entries and the batch size. Before sending the dataset, the number of training rounds and the number of batches contained in the training dataset and the test dataset in one training round are sent first.

[0066] Step S4: The client serializes the data, setting the first four bytes as the data length information and the subsequent bytes as training data. The dataset contains wireless cellular traffic values ​​from approximately 150 base stations in a certain area, covering the period from July 28, 2024 to August 25, 2024, with a data collection interval of 15 minutes. The wireless cellular traffic data is serialized into a byte stream to help preserve the complex data structure and quickly send it to the server.

[0067] Step S5: The client uses the TCP / IP communication protocol to quickly send the serialized data to the server;

[0068] Step S6: The client blocks and waits for the server to send a "receive complete" feedback message;

[0069] Step S7: Repeat steps S4 to S6 until the preset training termination conditions are met.

[0070] The specific implementation steps of the server in a storage-compute separation scenario include:

[0071] Step T1: The server uses a multi-process architecture, with each process controlling one GPU device.

[0072] Step T2: The server-side main process creates a dedicated data receiving thread responsible for receiving data sent by the client. After receiving the serialized data, this thread first parses the first four bytes to accurately obtain the length information of the training data. Then, it obtains the complete actual training data through a loop receiving method. The specific steps include:

[0073] Step T21: The server first parses the training data length information from the first four bytes of the received data, then receives the data in blocks and concatenates the data into a byte array;

[0074] Step T22: Deserialize the received byte stream data into wireless cellular traffic data;

[0075] Step T3: The server-side data receiving thread further divides the received data into multiple sub-blocks and stores them in an orderly manner in a shared queue. Specifically, the received sequence data is evenly divided according to the number of training processes so that each process can process approximately the same amount of sequence data, and the divided data is stored in a shared queue for the training processes to use.

[0076] Step T4: The server-side initialization of the large language model GPT-2 is performed. To make the GPT-2 model more effective at processing time series data, a spatiotemporal embedding layer, a two-stream attention layer, and a graph convolutional layer are designed to improve the GPT-2 model. The improved model architecture is as follows: Figure 3 As shown: The spatiotemporal embedding layer adds base station ID embedding and multi-scale temporal embedding to the word embedding and location embedding built into the GPT-2 model; the dual-stream attention layer splits the attention layer contained in each Transformer block into attention sub-layers and spatial attention sub-layers, and dynamically fuses them through a gating mechanism; the graph convolution enhancement layer adds a graph convolutional network layer after the attention layer to capture base station topology information and improve the ability to model spatial propagation patterns;

[0077] Step T5: The training process extracts data blocks from the queue in sequence for spatiotemporal awareness forward propagation. Multimodal feature fusion is performed through the spatiotemporal embedding layer to encode heterogeneous input information into the same feature space. Different types of information, such as numerical traffic data, discrete base station identifiers, and periodic time information, are fused. The optimal representation of each dimension is learned through the embedding layer to provide semantically rich input vectors for subsequent attention calculation.

[0078] Step T6: After uniformly encoding heterogeneous information through the spatiotemporal embedding layer, the temporal and spatial correlations of wireless cellular traffic are modeled through the dual-stream attention layer. Traffic patterns are captured through the temporal attention sub-layer, and the traffic correlations of functionally similar regions are identified through the spatial attention sub-layer. The two attention streams are dynamically weighted and fused through a gating mechanism.

[0079] Step T7: Enhance the topological information of the base station network through graph convolutional layers, apply graph convolution to the output of the attention layer to enhance spatial features, supplement the local spatial structure information that the attention mechanism may ignore, and improve the ability to model spatial propagation patterns.

[0080] Step T8: The feedforward neural network retains the original structure of GPT-2, uses a fully connected network and the GELU activation function, and inherits the powerful representation capabilities of the pre-trained model;

[0081] Step T9: Repeat steps T5 and T8 until the training termination condition is met.

[0082] After the entire model training is completed, the performance of the memory-compute separation architecture model also needs to be evaluated. The specific steps include:

[0083] Step W1: For a single batch of data, record a timestamp at the end of each batch of training, and subtract adjacent timestamps to obtain the training time of a batch;

[0084] Step W2: Record the time for forward propagation, back propagation, and gradient update for each batch of data as the computation time, and record the difference between the training time and computation time for a batch of data as the communication time;

[0085] Step W3: Record the training time, communication time, and computation time for each batch under both in-memory / in-memory separation and in-memory / integrated computing scenarios, and compare and plot the results. Figure 4 As shown.

[0086] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0087] Therefore, this invention provides a method for remote training of the large language model GPT-2 in a storage-compute separation scenario. It achieves physical separation of data storage and model training through a client-server architecture. The client is responsible for data serialization and transmission, while the server is responsible for data reception, deserialization, and multi-process distributed training. This method supports the parallel execution of data reception and training processes, and uses an improved GPT-2 model to enhance the modeling capability of spatiotemporal sequence data, thereby improving training efficiency and system resource utilization.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for remote training of a large language model GPT-2 in a storage-compute separation scenario, comprising a client system and a server system, wherein a communication connection is established between the client and the server using TCP / IP, a desktop computer acts as the client for storing and sending data, and remote computing resources act as the server for undertaking computing tasks, characterized in that, The specific steps are as follows: Step S1: The client establishes a communication connection with the server and sends data description information, which includes the training rounds, the number of training batches, and the number of test batches; Step S2: The client serializes the data and sends it to the server via the network. The serialized data includes the first four bytes indicating the data length and the subsequent actual training data. Step S3: The data receiving thread in the server's main process receives the serialized data, parses the first four bytes to determine the length of the data block, and stores the data in a shared queue; Step S4: The server uses multiple processes to train the model. Each training process extracts data blocks from the shared queue, performs word segmentation, and then trains the GPT-2 model in parallel. Step S5: Repeat steps S2 to S4 until the preset training termination condition is met; Step S6: Evaluate the performance of the trained GPT-2 model; In step S4, before training the model, the server also needs to establish a communication connection, select a model, create a data receiving thread, and run multiple training processes. Specific steps include: Step S41: Establish a communication connection. The server listens for communication connection requests on a specified port and establishes a communication connection with the client. Step S42: Construct a multi-process distributed parallel training model GPT-2 based on the generative pre-trained Transformer model. The GPT-2 model includes a word position embedding layer, a word embedding layer, 12 repeatedly stacked GPT blocks and a layer normalization layer. Each GPT block contains two layer normalization layers, an attention layer and a multilayer perceptron. The hidden layer dimension is 768. Step S43: Create multiple training processes. The server-side model training uses a distributed training method, creating multiple processes to control multiple graphics card devices to execute the same training task. After the multi-process training is completed, the model parameters are updated simultaneously. Step S44: Create a data receiving thread. Create a data receiving thread in the server's main process. The internal architecture of the GPT-2 model was improved, specifically as follows: The spatiotemporal embedding layer adds base station ID embedding and multi-scale temporal embedding to the word embedding and location embedding built into the GPT-2 model; The original attention layer of the GPT-2 model is improved into a dual-stream attention layer, which includes a temporal attention sub-layer and a spatial attention sub-layer, and is dynamically fused through a gating mechanism. A graph convolutional layer is added after the dual-stream attention layer to capture base station topology information.

2. The method for remote training of a large language model GPT-2 in a storage-compute separation scenario according to claim 1, characterized in that: In step S1, before the client sends data to the server, it determines the client-server communication port, establishes a communication connection, selects the training batch size and training dataset. Specific steps include: Step S11: Select a port number. When the client and server establish a communication connection, both ends need to select an available port for establishing the connection and transmitting data. You cannot select a system-reserved port number or a port number that has been occupied by an application. Step S12: Establish a communication connection. The client uses the temporary port allocated by the operating system to establish a communication connection with the specified port of the server. Step S13: Select the batch size based on the server-side computing resource hardware configuration.

3. The method for remote training of a large language model GPT-2 in a storage-compute separation scenario according to claim 1, characterized in that: In step S2, after the client establishes a communication connection with the server, it needs to serialize data and send training round description information. The specific steps include: Step S21: Serialize and send data, serialize the data into a byte stream and send the byte stream data; Step S22: Send training round description information. Calculate the batch number information of the training dataset and the test dataset based on the number of dataset entries and batch size. Before sending the dataset, first send the number of training rounds and the number of batches contained in the training dataset and the test dataset in one training round.

4. The method for remote training of a large language model GPT-2 in a storage-compute separation scenario according to claim 1, characterized in that: In step S3, the server-side data receiving thread performs the tasks of data reception, deserialization, and storage in the shared queue. The specific steps include: Step S31: The server first parses the training data length information from the first four bytes of the received data, then receives the data in blocks and concatenates the data into a byte array; Step S32: Deserialize the received byte stream data; Step S33: Divide the received sequence data evenly according to the number of training processes, and store the divided data in a shared queue for use by the training processes.

5. The method for remote training of a large language model GPT-2 in a storage-compute separation scenario according to claim 1, characterized in that: In step S43, the server-side training process for training the model includes retrieving data from the shared queue, word segmentation, model training, backpropagation, gradient aggregation, and parameter updates. Specific steps include: Step S431: Each training process retrieves data from the shared queue, determines whether there is data in the shared queue, and blocks and waits until there is data in the shared queue when the shared queue is empty. Step S432: Use a tokenizer to convert continuous data into a series of sub-unit tokens; Step S433: Train the model. Before training begins, clear the gradients of all parameters to zero. Use a sliding window mechanism to construct the source input and target output. The source input is mapped to a word embedding representation through the word embedding layer. The word embedding representation and the word position embedding representation are added element by element and then used as the model input. The model input is output after forward propagation. The model output is converted into a token representation after passing through the language head layer. Calculate the cross-entropy loss between the model output and the target output. Step S434: Calculate the gradient through backpropagation, and temporarily store the calculated gradient result in the grad attribute of the model parameters; Step S435: Aggregate gradients. Aggregate the gradient results obtained from multiple training processes using an average aggregation strategy, and distribute the aggregated results to each training process. Step S436: Update the model parameters using the AdamW adaptive moment estimator with separated weight decay.

6. The method for remote training of a large language model GPT-2 in a storage-compute separation scenario according to claim 1, characterized in that: In step S6, after the entire model training is completed, the performance of the memory-compute separation architecture model is evaluated. Specific steps include: Step S61: For a single batch of data, record a timestamp at the end of each batch of training, and subtract adjacent timestamps to obtain the training time of a batch; Step S62: Record the time for forward propagation, back propagation and gradient update for each batch of data as the computation time, and record the difference between the training time and computation time for a batch of data as the communication time; Step S63: Record the training time, communication time, and computation time for each batch under the scenarios of storage-computation separation and storage-computation integration, and draw a comparison graph.

Images