Neural network computing method and apparatus based on spatiotemporal balancing

Abstract

The application relates to a neural network computing method and device based on space-time balance, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: obtaining a re-computing strategy of a neural network to be trained; identifying a sending end flow segment and a receiving end flow segment in each training flow segment according to the re-computing strategy, and determining a first storage resource of each sending end flow segment and a second storage resource of each receiving end flow segment; and training the neural network to obtain a target neural network through each sending end flow segment according to the first storage resource and through each receiving end flow segment according to the second storage resource. The method can improve the training efficiency of the neural network.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for neural network computation based on spatiotemporal equilibrium. Background Technology

[0002] With the continuous improvement of data processing capabilities, neural network technology has also gradually developed rapidly. Neural networks are usually used as the inference layer of various models to help the models achieve functions such as reasoning, mapping, and prediction. For large language models, neural networks can help large language models achieve functions such as natural language processing, semantic analysis, and intent recognition. However, neural networks usually need to be trained to effectively achieve the corresponding functions.

[0003] Currently, neural networks can be trained using parallel computing to achieve functions such as natural language processing, semantic analysis, and intent recognition. However, current neural network training methods still suffer from insufficient training efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a spatiotemporal equalization-based neural network computing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the training efficiency of neural networks, addressing the aforementioned technical problems.

[0005] In a first aspect, this application provides a neural network computation method based on spatiotemporal equilibrium, comprising: obtaining a recomputation strategy for a neural network to be trained, wherein the recomputation strategy is used to characterize the storage resources and training time of each training pipeline segment, and each training pipeline segment is used to perform parallel computation on the neural network; identifying the sending pipeline segment and the receiving pipeline segment in each training pipeline segment according to the recomputation strategy, and determining the first storage resource of each sending pipeline segment and the second storage resource of each receiving pipeline segment; training the neural network according to the first storage resource in each sending pipeline segment and according to the second storage resource in each receiving pipeline segment to obtain a trained target neural network, wherein the target neural network is used for natural language processing.

[0006] In one embodiment, obtaining a recomputation strategy for a neural network to be trained includes: obtaining a parallel computing configuration of the neural network, the parallel computing configuration being used to characterize the computing and storage resources allocated to the neural network to be trained; and determining a recomputation strategy based on the parallel computing configuration.

[0007] In one embodiment, determining a recomputation strategy based on the parallel computing configuration includes: allocating each training pipeline segment to the neural network, and determining the storage resources and training time of each training pipeline segment based on the parallel computing configuration; and determining the recomputation strategy based on the storage resources and training time of each training pipeline segment.

[0008] In one embodiment, determining the storage resources and training time of each training pipeline segment based on the parallel computing configuration includes: determining a training time determination function based on the parallel computing configuration; using each computing power resource and storage resource as constraints, solving for the minimum value of the training time determination function to obtain the storage resources and training time of each training pipeline segment.

[0009] In one embodiment, identifying the sender pipeline and receiver pipeline in each pipeline according to the recomputation strategy includes: for each training pipeline, comparing the storage resources with the local storage resources corresponding to the training pipeline; if the storage resources are greater than the local storage resources, determining the training pipeline as the sender pipeline; if the storage resources are less than or equal to the local storage resources, determining the training pipeline as the receiver pipeline.

[0010] In one embodiment, determining the first storage resource of each transmitting pipeline segment and the second storage resource of each receiving pipeline segment includes: summing the storage resources corresponding to each transmitting pipeline segment to obtain the first candidate storage resource of each transmitting pipeline segment; summing the storage resources corresponding to each receiving pipeline segment to obtain the second candidate storage resource of each receiving pipeline segment; if the difference between the first candidate storage resource and the second candidate storage resource is greater than the difference threshold, then determining the communication parameters of each transmitting pipeline segment or each receiving pipeline segment; determining each first storage resource and each second storage resource based on each communication parameter, the first candidate storage resource, and the second candidate storage resource; the communication parameters are determined based on the shortest communication distance and the parallel communication load.

[0011] In one embodiment, each first storage resource and each second storage resource are determined based on each communication parameter, a first candidate storage resource, and a second candidate storage resource, including any one of the following: if the first storage resource is greater than the second storage resource, then the transmitting pipeline segment with the smallest communication parameter among the transmitting pipeline segments is updated as the receiving pipeline segment, thus obtaining each first storage resource and each second storage resource; if the second storage resource is greater than the first storage resource, then the receiving pipeline segment with the smallest communication parameter among the receiving pipeline segments is updated as the transmitting pipeline segment, thus obtaining each first storage resource and each second storage resource.

[0012] Secondly, this application also provides a neural network computing device based on spatiotemporal equalization, comprising: a strategy acquisition module, used to acquire a recomputation strategy for a neural network to be trained, the recomputation strategy being used to characterize the storage resources and training time of each training pipeline segment, each training pipeline segment being used to perform parallel computation on the neural network; an identification module, used to identify the sending pipeline segment and the receiving pipeline segment in each training pipeline segment according to the recomputation strategy, and to determine the first storage resource of each sending pipeline segment and the second storage resource of each receiving pipeline segment; and a training module, used to train the neural network according to the first storage resource through each sending pipeline segment and according to the second storage resource through each receiving pipeline segment to obtain a trained target neural network, the target neural network being used for natural language processing.

[0013] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect.

[0014] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0016] The aforementioned spatiotemporal balanced neural network computation method, apparatus, computer device, computer-readable storage medium, and computer program product, for the neural network to be trained, first obtain the recomputation strategy of the neural network to be trained. The recomputation strategy is used to characterize the storage resources and training time of each training pipeline segment. The training pipeline segment is used to perform parallel computation on the neural network. According to the recomputation strategy, the sending pipeline segment and the receiving pipeline segment in each training pipeline segment are identified, and the first storage resource of each sending pipeline segment and the second storage resource of each receiving pipeline segment are determined. By using each sending pipeline segment based on the first storage resource and each receiving pipeline segment based on the second storage resource, the neural network is trained to obtain the target neural network for natural speech processing. This achieves unified optimization of recomputation and training, solves the problems of recomputation overhead and storage imbalance, and improves the training efficiency of the neural network. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an application environment diagram of a neural network computation method based on spatiotemporal equilibrium in one embodiment;

[0019] Figure 2 This is a flowchart illustrating a neural network computation method based on spatiotemporal equilibrium in one embodiment;

[0020] Figure 3 This is a flowchart illustrating step 201 in one embodiment;

[0021] Figure 4 This is a flowchart illustrating step 302 in one embodiment;

[0022] Figure 5 This is a flowchart illustrating the steps for determining the storage resources and training time of each training pipeline segment based on the parallel computing configuration in one embodiment.

[0023] Figure 6 This is a flowchart illustrating step 202 in one embodiment;

[0024] Figure 7 This is a flowchart illustrating step 202 in another embodiment;

[0025] Figure 8 This is a flowchart illustrating step 703 in another embodiment;

[0026] Figure 9 This is a flowchart illustrating a neural network computation method based on spatiotemporal equilibrium in another embodiment;

[0027] Figure 10 This is a block diagram of a neural network computing device based on spatiotemporal equalization in one embodiment;

[0028] Figure 11 This is a schematic diagram of data interaction in a neural network computation method based on spatiotemporal equilibrium in one embodiment;

[0029] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0032] The spatiotemporal equilibrium-based neural network computation method provided in this application can be applied to, for example... Figure 1 The application environment shown includes at least a training device 101 and a neural network 102 to be trained.

[0033] The training device 101 is used to acquire the recomputation strategy of the neural network to be trained, identify the sending-end pipeline and receiving-end pipeline in each training pipeline according to the recomputation strategy, determine the first storage resource of each sending-end pipeline and the second storage resource of each receiving-end pipeline, and train the neural network 102 by each sending-end pipeline based on the first storage resource and by each receiving-end pipeline based on the second storage resource. The training device 101 can be a server, which can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0034] The neural network 102 to be trained is used to train according to the instructions of the training device 101 to obtain the trained target neural network. The architecture of the neural network 102 can be at least one of the following: recurrent neural network, long short-term memory network, gated recurrent unit, convolutional architecture, cue word architecture, encoder-decoder architecture, and hybrid model. The neural network to be trained can be deployed in a server, which can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0035] In real-world scenarios, neural network training typically involves three stages: forward computation, backpropagation, and gradient update. Backpropagation and gradient update usually require input activations or intermediate activations generated during the forward computation stage. Therefore, intermediate results generated during the forward stage often need to be temporarily stored until the corresponding backpropagation is complete. For large-scale neural network models, the large number of intermediate activations and their significant storage requirements can lead to high GPU memory overhead, thus limiting the training scale and efficiency.

[0036] In related technologies, multi-device parallel computing techniques are typically required to support the training of large-scale neural network models. Common approaches include tensor parallelism and pipelined parallelism. Tensor parallelism involves dividing the parameter tensors or computational tasks of the same model layer across multiple devices, allowing multiple devices to collaboratively complete the computation of the same layer. This approach reduces the parameter storage pressure and computational load on a single device. Pipeline parallelism involves dividing multiple consecutive model layers in a neural network into multiple devices for execution. Such a device is called a pipeline. Input activations flow sequentially through each pipeline, and the output of each pipeline corresponds to an intermediate activation and is passed to the next pipeline until the output activation is generated. This approach can improve training throughput and reduce the model size that a single device can handle through collaborative execution of multiple devices.

[0037] For pipelined parallel training, a training batch is typically further divided into multiple micro-batches and executed using a 1F1B scheduling method. 1F1B scheduling means that after the pipeline reaches a steady state, each pipeline alternately executes one forward process and one backward process. During pipelined training, there is a significant imbalance in storage usage across different devices, with earlier devices having higher storage usage and later devices having lower storage usage. This is because intermediate activations generated by forward computation usually need to be retained until the corresponding backward propagation stage before being released, and earlier devices need to retain more intermediate activations corresponding to micro-batches. For a pipelined parallel system of stage p, the i-th device typically needs to retain intermediate activations corresponding to approximately pi micro-batches, thus more easily leading to higher storage pressure.

[0038] Furthermore, to reduce the storage overhead of intermediate activations in the forward propagation phase, existing training systems typically employ recomputation techniques. Recomputation refers to saving only a portion of the activation checkpoints during the forward propagation phase, and then re-executing the corresponding forward computations for the unsaved intermediate activations during the backpropagation phase to recover the data required for gradient calculation. This reduces activation storage during training. The backpropagation time includes not only the original computational overhead but also the additional execution time corresponding to the operators involved in the recomputation. Therefore, while recomputation reduces storage usage, it increases the execution latency of the backpropagation phase. In pipelined parallel training, if the recomputation tasks configured in different pipeline stages differ, the execution time of each pipeline stage will become unbalanced. This can cause some devices in the pipeline to wait for other devices to complete their computations, creating additional pipeline bubbles and reducing training efficiency.

[0039] To address this, this application first obtains the recomputation strategy of the model to be trained, and identifies each sending-end pipeline and receiving-end pipeline in the recomputation strategy based on local storage. For each training pipeline, it determines the first storage resource of each sending-end pipeline and the second storage resource of each receiving-end pipeline. The neural network is then trained using the first storage resource of each sending-end pipeline and the second storage resource of each receiving-end pipeline to obtain the trained target neural network. In this way, while satisfying global storage constraints, the recomputation overhead of different pipelines can be relatively evenly distributed, thereby reducing execution time imbalance and pipeline bubbles caused by unbalanced recomputation. At the same time, by establishing a collaborative storage relationship between the sending and receiving ends, overflow activations in pipelines with high storage pressure are transferred to pipelines with remaining storage space for temporary storage, thereby reducing the additional recomputation overhead of local pipelines, minimizing system training latency, and improving training efficiency.

[0040] In one exemplary embodiment, such as Figure 2 As shown, a neural network computation method based on spatiotemporal equalization is provided, which is then applied to... Figure 1 The following steps are used as an example of training device 101, including steps 201 to 203.

[0041] Step 201: Obtain the recomputation strategy for the neural network to be trained.

[0042] During implementation, the training device first acquires the neural network to be trained and its corresponding recomputation strategy. The recomputation strategy characterizes the storage resources and training time of each training pipeline stage, which performs parallel computation on the neural network. A training pipeline stage is an execution unit obtained by dividing the model into layers, sublayers, or operator groups in a parallel pipeline. Each pipeline stage is typically mapped to one or more computing devices, responsible for completing the forward computation and backward propagation of its corresponding part.

[0043] In acquiring the recomputation strategy, the training device can determine the recomputation strategy based on the parallel computing configuration, or based on the model training configuration, or based on both the parallel computing configuration and the model training configuration. The model training configuration can represent the model's training objectives, such as accuracy or learning rate.

[0044] Furthermore, multiple training samples of the neural network to be trained and the corresponding labels of each training sample can be obtained. The neural network can be a neural network used for natural speech processing, which can realize functions such as semantic recognition, intent recognition, data generation, and data analysis.

[0045] In practical applications, training devices can also determine recomputation strategies through activation checkpoints. The training device can calculate the available storage for activation checkpoints based on the parallel computing configuration and / or model training configuration, and dynamically plan and solve the recomputation strategy. Activation checkpoints refer to intermediate activation data or intermediate tensors selected and saved during neural network training to recover intermediate results in subsequent backpropagation.

[0046] Step 202: Identify the transmitter pipeline and receiver pipeline in each training pipeline according to the recomputation strategy, and determine the first storage resource of each transmitter pipeline and the second storage resource of each receiver pipeline.

[0047] During implementation, the training equipment extracts each training pipeline segment included in the recalculation strategy. Based on the storage resources and training time of each training pipeline segment, it divides each training pipeline segment into a sending pipeline segment and a receiving pipeline segment. Furthermore, based on the storage resources and training time of each training pipeline segment, it determines the first storage resource for each sending pipeline segment and the second storage resource for each receiving pipeline segment. Specifically, the sending pipeline segment refers to the pipeline segment that, during checkpoint recalculation and storage allocation, needs to send overflowing checkpoints to other pipeline segments or other devices for storage when local storage resources are insufficient. The sending pipeline segment can send its forward calculation results and / or reverse calculation results to the corresponding receiving pipeline segment. The receiving pipeline segment is a pipeline segment or device with remaining storage resources during checkpoint recalculation and storage allocation, capable of receiving and storing checkpoints from the sending pipeline segment. The receiving pipeline segment can receive the forward calculation results and / or reverse calculation results sent by the sending pipeline segment.

[0048] During execution, the training device can divide the training pipeline into sender and receiver pipelines based on the storage resources and training time of each training pipeline, as well as the local storage resources allocated by the training device for each training pipeline. Furthermore, based on the storage resources and training time of each training pipeline, and the local storage resources allocated by the training device for each training pipeline, it determines the first storage resource for each sender pipeline and the second storage resource for each receiver pipeline. In addition, the training device can input the storage resources and training time of each training pipeline into a specific algorithm / model for data processing to obtain the first storage resource for each sender pipeline and the second storage resource for each receiver pipeline. The training configuration may include the local storage resources allocated by the training device for each training pipeline.

[0049] Step 203: Train the neural network using the first storage resource through each sending pipeline segment and the second storage resource through each receiving pipeline segment to obtain the trained target neural network.

[0050] During implementation, the training device performs forward and / or backward computations based on the first storage resource through the transmitting pipeline to obtain the computation results, which are then transmitted to the receiving end. Correspondingly, the receiving pipeline receives the computation results sent by the transmitting pipeline, processes the data, and repeats the above steps until the neural network training meets the completion conditions, resulting in the target neural network. For multiple transmitting and receiving pipelines, both can simultaneously receive data, perform computations, and output data. The order / label of the pipelines can then determine whether it is a transmitting or receiving pipeline. Specific implementation methods can be found in relevant technologies and will not be elaborated upon here.

[0051] During execution, the training device can determine the training completion conditions according to the model training configuration. If the training completion conditions are met, the training of the neural network to be trained is stopped, and the target neural network is obtained after training is completed. The target neural network is used for natural speech processing. The training completion conditions can be that the number of training rounds reaches a preset number, or that the loss converges.

[0052] In the aforementioned spatiotemporal equalization-based neural network computation method, for the neural network to be trained, the recomputation strategy of the neural network to be trained is first obtained. The recomputation strategy is used to characterize the storage resources and training time of each training pipeline segment. The training pipeline segment is used to perform parallel computation on the neural network. According to the recomputation strategy, the sending pipeline segment and the receiving pipeline segment in each training pipeline segment are identified, and the first storage resource of each sending pipeline segment and the second storage resource of each receiving pipeline segment are determined. The neural network is trained by each sending pipeline segment based on the first storage resource and by each receiving pipeline segment based on the second storage resource to obtain the target neural network for natural speech processing. This achieves unified optimization of recomputation and training, solves the problems of recomputation overhead and storage imbalance, and improves the training efficiency of the neural network.

[0053] Based on the above exemplary embodiment, the following provides a spatiotemporal equalization-based neural network computation method in one or more exemplary embodiments, which can be applied to... Figure 1 The following explanation will be given using the training equipment as an example, and will include the following content.

[0054] In the process of obtaining the recomputation strategy, the recomputation strategy can be determined based on the parallel computing configuration; one optional implementation method provided is as follows: Figure 3 As shown, step 201 includes steps 301 to 302:

[0055] Step 301: Obtain the parallel computing configuration of the neural network.

[0056] During implementation, the training device acquires the parallel training configuration of the neural network to be trained. The parallel computing configuration represents the computing and storage resources allocated to the neural network. The parallel training configuration can be a training configuration received by the training device from an external device, or it can be preset configuration information read by the training device. In this application, the parallel computing configuration may include at least one of tensor parallelism, pipeline parallelism, and pipeline storage capacity.

[0057] Step 302: Determine the recomputation strategy based on the parallel computing configuration.

[0058] During implementation, the training device parses the parallel computing configuration and determines the recomputation strategy based on at least one of the tensor parallelism, pipeline parallelism, and pipeline storage capacity included in the parallel computing configuration.

[0059] During execution, the training device can perform operator-level modeling of the neural network to be trained based on the given tensor parallelism, pipeline parallelism and / or model configuration. It can divide the network into multiple basic operators based on the computation graph of the neural network to be trained, and label each operator with its computation type, forward computation time, back propagation time and storage overhead required to save intermediate activation checkpoints, so as to obtain the candidate recomputation strategies required for subsequent dynamic programming and their corresponding time benefits.

[0060] One optional implementation provided in this application avoids introducing more pipeline bubbles in the warm-up stage by balancing the execution time of the stabilization phase through a recalculation strategy and pipeline segmentation, thereby improving the usability of the recalculation strategy.

[0061] In determining the recomputation strategy based on the parallel computing configuration, the training device can determine the storage resources and training time of each training pipeline segment, and thus determine the recomputation strategy; in one optional implementation provided in this application, such as Figure 4 As shown, step 302 includes steps 401 to 402:

[0062] Step 401: Allocate training pipeline segments to the neural network and determine the storage resources and training time for each training pipeline segment based on the parallel computing configuration.

[0063] During implementation, the training equipment can allocate each training pipeline segment to the neural network according to the model training configuration, and determine the storage resources and training time of each training pipeline segment according to the parallel computing configuration.

[0064] During the allocation of training pipeline segments, the training device can determine the search range of dynamic programming, and label each operator with its computation type, forward computation time, backpropagation time, and storage overhead required to save intermediate activation checkpoints according to the search range, thus obtaining each training pipeline segment.

[0065] In determining storage resources and training time, the training device can jointly search for storage allocation methods and recomputation schemes for each pipeline under the global storage constraints shared by multiple pipelines. It can use the overall storage resources corresponding to each training pipeline as constraints and determine the storage resources and training time of each training pipeline based on the parallel computing configuration.

[0066] Step 402: Determine the recomputation strategy based on the storage resources and training time of each training pipeline segment.

[0067] During implementation, the training equipment constructs a recomputation strategy for the neural network to be trained based on the storage resources and training time of each training pipeline segment.

[0068] In one optional implementation provided by this application, the shared storage resources of each training pipeline are used as constraints to jointly search the storage allocation method and recomputation scheme of each pipeline, thereby improving the reliability and pertinence of the recomputation strategy.

[0069] In determining storage resources and training time, the training device can construct a function that uses storage resources as a constraint to calculate the minimum training time; one optional implementation provided in this application is as follows: Figure 5 As shown, determining the storage resources and training time for each training pipeline segment based on the parallel computing configuration includes steps 501 to 502:

[0070] Step 501: Determine the training time determination function based on the parallel computing configuration.

[0071] During implementation, the training equipment constructs a training time determination function based on the data included in the parallel computing configuration.

[0072] During the execution process, the first There are several pipeline segments, each allocated a storage capacity of 1. Then the remaining The storage capacity of that size continues to be allocated to subsequent pipeline stages. At this point, starting from the first... The bottleneck execution time from the first flow segment to the last flow segment can be expressed as formula (1):

[0073] Formula (1);

[0074] in, This indicates the minimum bottleneck execution time that subsequent pipeline stages can achieve with the remaining storage capacity; Indicates the first Forward computation time for each flow segment; Indicates the first The reverse propagation time of each flow segment; Indicates the first The basic execution time of each flow segment during the stable phase of the 1F1B flow scheduling; This indicates that the water flow section is allocated to The time gains from employing appropriate recomputation strategies when considering storage capacity size. Indicates the first The execution time of each flow segment after adopting the corresponding recalculation strategy.

[0075] Step 502: Using each computing power resource and storage resource as constraints, the minimum value of the training time determination function is solved to obtain the storage resources and training time of each training pipeline segment.

[0076] During implementation, the training equipment uses the computing and storage resources as constraints to solve for the minimum value of the training time determination function, thereby obtaining the storage resources and training time for each training pipeline segment.

[0077] During execution, the minimum bottleneck time is recorded as follows: Record the corresponding storage allocation results as See formula (2):

[0078] Formula (2).

[0079] One optional implementation provided in this application uses a recomputation strategy to employ recomputation schemes of different granularities and intensities in each pipeline segment, thereby balancing the execution time of each pipeline segment as much as possible, reducing pipeline bubbles caused by uneven distribution of recomputation overhead, and thus improving the training efficiency of the neural network.

[0080] In determining the transmitting and receiving pipeline segments within each pipeline segment, the training device can determine the transmitting and receiving pipeline segments based on storage resources and the local storage resources corresponding to the training pipeline segments; In one optional implementation provided in this application, such as... Figure 6 As shown, step 202 includes steps 601 to 602 or step 603:

[0081] Step 601: For each training pipeline segment, compare the storage resources with the local storage resources corresponding to the training pipeline segment.

[0082] During implementation, for each training pipeline segment, the training device compares the storage resources with the local storage resources corresponding to the training pipeline segment.

[0083] During execution, the training device compares the storage resources of each training pipeline segment with the local storage resources corresponding to the training pipeline segment. If the storage resources are greater than the local storage resources, it indicates that the pipeline segment still has local storage overflow after adopting the recompute strategy, so it is identified as the sending pipeline segment. If the storage resources are less than or equal to the local storage resources, it indicates that the pipeline segment has free capacity that can be used for collaborative storage, so it is added to the receiving pipeline segment.

[0084] Step 602: If the storage resources are greater than the local storage resources, determine the training pipeline segment as the sending pipeline segment.

[0085] During implementation, if the storage resources are greater than the local storage resources, it indicates that the pipeline segment still has local storage overflow after adopting the recompute strategy, and the training pipeline segment is determined to be the sending pipeline segment.

[0086] Step 603: If the storage resources are less than or equal to the local storage resources, determine the training pipeline as the receiving pipeline.

[0087] During implementation, if the storage resources are less than or equal to the local storage resources, it indicates that the pipeline segment has free capacity available for collaborative storage, and the training pipeline segment is determined to be the receiving pipeline segment.

[0088] One optional implementation provided in this application determines the transmitting end pipeline and the receiving end pipeline by comparing storage resources. In this way, each transmitting end pipeline and each receiving end pipeline are accurately determined based on storage resources, thereby improving the reliability of identification.

[0089] In determining the first storage resources of each transmitting pipeline segment and the second storage resources of each receiving pipeline segment, the training device can compare the total storage resources of each transmitting pipeline segment with the total storage resources of each receiving pipeline segment, and can further combine their respective communication parameters to determine each first storage resource and each second storage resource; in one optional implementation provided by this application, such as Figure 7 As shown, step 202 includes steps 701 to 703:

[0090] Step 701: Sum the storage resources corresponding to each sending pipeline segment to obtain the first candidate storage resources for each sending pipeline segment.

[0091] During implementation, the training equipment sums the storage resources corresponding to each transmitter pipeline segment to obtain the first candidate storage resource for each transmitter pipeline segment.

[0092] Step 702: Summation of the storage resources corresponding to each receiver pipeline segment is performed to obtain the second candidate storage resources for each receiver pipeline segment.

[0093] During implementation, the training equipment sums up the storage resources corresponding to each receiver pipeline segment to obtain the second candidate storage resources for each receiver pipeline segment.

[0094] Step 703: If the difference between the first candidate storage resource and the second candidate storage resource is greater than the difference threshold, then determine the communication parameters of each sending pipeline segment or each receiving pipeline segment, and determine each first storage resource and each second storage resource based on each communication parameter, the first candidate storage resource and the second candidate storage resource.

[0095] In this application, communication parameters are parameters used to characterize communication consumption / communication cost. Communication parameters may include the shortest communication distance between adjacent pipeline segments and / or the pipeline parallel communication load between adjacent pipeline segments. Communication parameters are determined based on the shortest communication distance and the parallel communication load.

[0096] During implementation, if the difference between the first candidate storage resource and the second candidate storage resource is greater than the difference threshold, the training device determines the communication parameters of each transmitting pipeline segment or each receiving pipeline segment, and determines each first storage resource and each second storage resource based on each communication parameter, the first candidate storage resource and the second candidate storage resource.

[0097] During execution, communication parameters can be used as constraints to determine each first storage resource and each second storage resource based on the first candidate storage resource and the second candidate storage resource.

[0098] In one optional implementation, the overflow storage capacity of the sending end and the remaining storage capacity of the receiving end can be sorted in descending order, so that the sending end with greater storage pressure is processed first, and the receiving end with greater available capacity is utilized first, thereby improving the feasibility of storage resource allocation.

[0099] In determining each first storage resource and each second storage resource, resources from the sending pipeline can be transferred to the receiving end, and vice versa; in one optional implementation provided by this application, such as Figure 8 As shown, step 703 includes either step 801 or step 802:

[0100] Step 801: If the first storage resource is greater than the second storage resource, then update the sender pipeline segment with the smallest communication parameters in each sender pipeline segment to the receiver pipeline segment, thereby obtaining each first storage resource and each second storage resource.

[0101] During implementation, if the first storage resource is greater than the second storage resource, the training device can sort the pipeline segments of each transmitting end according to the communication parameters, update the pipeline segment of the transmitting end with the smallest communication parameters to the pipeline segment of the receiving end, and obtain each first storage resource and each second storage resource.

[0102] The process for determining the communication parameters can be found in formula (3):

[0103] Formula (3);

[0104] in, Indicates the shortest communication distance between adjacent water flow sections. This indicates the parallel communication load between adjacent flow sections. Indicates the sending end With the receiving end The shortest communication distance between them This indicates the transmission load generated between the two due to the activation of checkpoint collaborative storage; This represents the communication path conflict penalty coefficient.

[0105] Step 802: If the second storage resource is greater than the first storage resource, then update the receiver pipeline segment with the smallest communication parameters in each receiver pipeline segment to the sender pipeline segment, thereby obtaining each first storage resource and each second storage resource.

[0106] During implementation, if the first storage resource is greater than the second storage resource, the training device can sort the pipelines of each receiver according to the communication parameters, update the receiver pipeline with the smallest communication parameters in each receiver pipeline to the transmitter pipeline, and obtain each first storage resource and each second storage resource.

[0107] It should be noted that the update process may differ between the sender and receiver. Firstly, the sender's required overflow storage may be deducted from the receiver's capacity, and the receiver with remaining capacity is reinserted into the candidate queue to continue participating in pairing with other senders. Secondly, if the receiver's capacity is insufficient, all of its remaining capacity is used to handle some of the sender's overflow activation checkpoints, and the next receiver with lower communication costs is selected from the candidate queue until the sender's overflow storage requirement is met.

[0108] One optional implementation provided in this application, through collaborative storage pairing between the sending pipeline and the receiving pipeline, not only completes time-balanced recomputation, but also alleviates local storage overflow through spatial balance pairing, enabling pipelined parallel training to maintain low recomputation and communication overhead under memory-constrained conditions, thereby further improving the training efficiency of the neural network.

[0109] In one embodiment, see Figure 9 The document illustrates a flowchart of a spatiotemporal equilibrium-based neural network computation method provided in an embodiment of this application. This spatiotemporal equilibrium-based neural network computation method can be applied to... Figure 1 The training equipment shown. (As indicated) Figure 9 As shown, the neural network computation method based on spatiotemporal equilibrium may include the following steps:

[0110] Step 901: Obtain the parallel computing configuration of the neural network to be trained.

[0111] Step 902: Determine the training time determination function based on the parallel computing configuration.

[0112] Step 903: Using each computing power resource and storage resource as constraints, the minimum value of the training time determination function is solved to obtain the storage resources and training time of each training pipeline segment.

[0113] Step 904: Determine the recomputation strategy based on the storage resources and training time of each training pipeline segment.

[0114] Step 905: For each training pipeline segment, compare the storage resources with the local storage resources corresponding to the training pipeline segment.

[0115] Step 906: Determine each sending pipeline segment and each receiving pipeline segment based on the comparison results of storage resources and local storage resources.

[0116] Step 907: Summate the storage resources corresponding to each sending pipeline segment to obtain the first candidate storage resources for each sending pipeline segment.

[0117] Step 908: Summation of the storage resources corresponding to each receiver pipeline segment is performed to obtain the second candidate storage resources for each receiver pipeline segment.

[0118] Step 909: Determine each first storage resource and each second storage resource based on the difference between the first candidate storage resource and the second candidate storage resource, the communication parameters of each transmitting pipeline segment or each receiving pipeline segment, and the first candidate storage resource and the second candidate storage resource.

[0119] Step 910: The target neural network is trained by each transmitting pipeline according to the first storage resource and by each receiving pipeline according to the second storage resource.

[0120] It should be noted that the target neural network type is not limited to a specific model structure. Besides the Transformer model, this invention is also applicable to other neural network training scenarios with requirements for forward computation, backpropagation, and intermediate activation storage. This application balances the execution time of each pipeline stage in the time dimension through a differentiated recomputation strategy under global storage constraints, and balances the storage pressure of activation checkpoints in the spatial dimension through cross-pipeline collaborative storage pairing, thereby reducing pipeline bubbles and local storage overflows under memory-constrained conditions. This application proposes a spatiotemporal dual-dimensional balancing mechanism, namely, reducing and balancing the recomputation execution overhead through time balancing, and redistributing the storage pressure of activation checkpoints through spatial balancing, thereby improving the scalability and execution efficiency of pipelined parallel training in memory-constrained large model training scenarios.

[0121] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0122] Based on the same inventive concept, this application also provides a spatiotemporal equilibrium-based neural network computing device for implementing the spatiotemporal equilibrium-based neural network computing method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the spatiotemporal equilibrium-based neural network computing device provided below can be found in the limitations of the spatiotemporal equilibrium-based neural network computing method described above, and will not be repeated here.

[0123] In one exemplary embodiment, such as Figure 10 As shown, a spatiotemporal equalization-based neural network computing device is provided, comprising: a policy acquisition module 1001, an identification module 1002, and a training module 1003, wherein: the policy acquisition module is used to acquire the recomputation strategy of the neural network to be trained, the recomputation strategy being used to characterize the storage resources and training time of each training pipeline segment, and each training pipeline segment being used to perform parallel computation on the neural network; the identification module is used to identify the sending pipeline segment and the receiving pipeline segment in each training pipeline segment according to the recomputation strategy, and to determine the first storage resource of each sending pipeline segment and the second storage resource of each receiving pipeline segment; the training module is used to train the neural network according to the first storage resource of each sending pipeline segment and according to the second storage resource of each receiving pipeline segment to obtain a trained target neural network, the target neural network being used for natural language processing.

[0124] In one embodiment, the strategy acquisition module 1001 is further configured to acquire the parallel computing configuration of the neural network, which represents the computing and storage resources allocated to the neural network to be trained; and determine the recomputation strategy based on the parallel computing configuration.

[0125] In one embodiment, the strategy acquisition module 1001 is further configured to allocate each training pipeline segment to the neural network, and determine the storage resources and training time of each training pipeline segment according to the parallel computing configuration; and determine the recomputation strategy based on the storage resources and training time of each training pipeline segment.

[0126] In one embodiment, the strategy acquisition module 1001 is further configured to determine the training time determination function based on the parallel computing configuration; and to solve for the minimum value of the training time determination function by taking each computing power resource and storage resource as constraints, thereby obtaining the storage resources and training time of each training pipeline segment.

[0127] In one embodiment, the identification module 1002 is further configured to compare the storage resources and the local storage resources corresponding to the training pipeline segment for each training pipeline segment; if the storage resources are greater than the local storage resources, the training pipeline segment is determined to be a sending pipeline segment; if the storage resources are less than or equal to the local storage resources, the training pipeline segment is determined to be a receiving pipeline segment.

[0128] In one embodiment, the identification module 1002 is further configured to sum the storage resources corresponding to each transmitting pipeline segment to obtain a first candidate storage resource for each transmitting pipeline segment; sum the storage resources corresponding to each receiving pipeline segment to obtain a second candidate storage resource for each receiving pipeline segment; if the difference between the first candidate storage resource and the second candidate storage resource is greater than the difference threshold, then the communication parameters of each transmitting pipeline segment or each receiving pipeline segment are determined, and each first storage resource and each second storage resource are determined according to each communication parameter, the first candidate storage resource and the second candidate storage resource, wherein the communication parameters are determined according to the shortest communication distance and the parallel communication load.

[0129] In one embodiment, the identification module 1002 is further configured to, if the first storage resource is greater than the second storage resource, update the transmitter pipeline segment with the smallest communication parameter among the transmitter pipeline segments to the receiver pipeline segment, thereby obtaining each first storage resource and each second storage resource; if the second storage resource is greater than the first storage resource, update the receiver pipeline segment with the smallest communication parameter among the receiver pipeline segments to the transmitter pipeline segment, thereby obtaining each first storage resource and each second storage resource.

[0130] The modules in the aforementioned spatiotemporal equalization-based neural network computing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0131] In one embodiment, another embodiment is also provided, such as Figure 11As shown, the training device first performs a globally balanced recalculation. Under the constraint of globally available storage, it allocates relatively balanced recalculation tasks to different pipeline stages. This allows some activations that originally needed to be saved to be recovered through recalculation during the reverse engineering phase, thereby reducing the amount of activations saved in pipeline stages with high storage pressure and ensuring that the overall activation storage requirements meet global storage requirements. Simultaneously, by balancing the recalculation overhead of different pipeline stages, it reduces the execution time imbalance during the reverse engineering phase, thus reducing pipeline bubbles introduced as a result. The sending and receiving ends are further identified. Pipeline stages with overflow activations after recalculation are designated as sending ends; pipeline stages with remaining storage space are designated as receiving ends. Subsequently, the overflow activations in the sending end are transferred to the receiving end for temporary storage and then sent back to the sending end when the corresponding reverse engineering occurs, maintaining the mathematical correctness of the training process. Therefore, in this embodiment, global balanced recomputation first balances the recomputation overhead of different pipeline sections while satisfying global storage constraints, thereby reducing pipeline bubbles caused by unbalanced recomputation. Then, local storage overflow is resolved through collaborative storage between the sending and receiving ends, thereby improving global storage utilization and improving storage balance between pipeline sections.

[0132] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores training data for the neural network. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a spatiotemporal equalization-based neural network computation method.

[0133] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: obtaining a recomputation strategy for a neural network to be trained, the recomputation strategy being used to characterize the storage resources and training time of each training pipeline segment, each training pipeline segment being used to perform parallel computation on the neural network; identifying the sending pipeline segment and the receiving pipeline segment in each training pipeline segment according to the recomputation strategy, and determining the first storage resource of each sending pipeline segment and the second storage resource of each receiving pipeline segment; training the neural network according to the first storage resource by each sending pipeline segment and according to the second storage resource by each receiving pipeline segment to obtain a trained target neural network, the target neural network being used for natural language processing.

[0135] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a parallel computing configuration of the neural network, the parallel computing configuration being used to characterize the computing and storage resources allocated to the neural network to be trained; and determining a recomputation strategy based on the parallel computing configuration.

[0136] In one embodiment, when the processor executes the computer program, it further performs the following steps: allocating training pipelines to the neural network and determining the storage resources and training time of each training pipeline according to the parallel computing configuration; and determining a recomputation strategy based on the storage resources and training time of each training pipeline.

[0137] In one embodiment, when the processor executes the computer program, it also performs the following steps: determining a training time determination function based on the parallel computing configuration; using each computing power resource and storage resource as constraints, solving for the minimum value of the training time determination function to obtain the storage resources and training time of each training pipeline segment.

[0138] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each training pipeline segment, it compares the storage resources with the local storage resources corresponding to the training pipeline segment; if the storage resources are greater than the local storage resources, it determines the training pipeline segment as the sending pipeline segment; if the storage resources are less than or equal to the local storage resources, it determines the training pipeline segment as the receiving pipeline segment.

[0139] In one embodiment, when the processor executes the computer program, it further performs the following steps: summing the storage resources corresponding to each transmitting pipeline segment to obtain a first candidate storage resource for each transmitting pipeline segment; summing the storage resources corresponding to each receiving pipeline segment to obtain a second candidate storage resource for each receiving pipeline segment; if the difference between the first candidate storage resource and the second candidate storage resource is greater than a difference threshold, then determining the communication parameters of each transmitting pipeline segment or each receiving pipeline segment, and determining each first storage resource and each second storage resource based on each communication parameter, the first candidate storage resource, and the second candidate storage resource, wherein the communication parameters are determined based on the shortest communication distance and the parallel communication load.

[0140] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the first storage resource is greater than the second storage resource, then the transmitting pipeline segment with the smallest communication parameter among the transmitting pipeline segments is updated to the receiving pipeline segment, thereby obtaining each first storage resource and each second storage resource; if the second storage resource is greater than the first storage resource, then the receiving pipeline segment with the smallest communication parameter among the receiving pipeline segments is updated to the transmitting pipeline segment, thereby obtaining each first storage resource and each second storage resource.

[0141] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: obtaining a recomputation strategy for a neural network to be trained, the recomputation strategy being used to characterize the storage resources and training time of each training pipeline segment, each training pipeline segment being used to perform parallel computation on the neural network; identifying the sending pipeline segment and the receiving pipeline segment in each training pipeline segment according to the recomputation strategy, and determining the first storage resource of each sending pipeline segment and the second storage resource of each receiving pipeline segment; training the neural network according to the first storage resource by each sending pipeline segment and according to the second storage resource by each receiving pipeline segment to obtain a trained target neural network, the target neural network being used for natural language processing.

[0142] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a parallel computing configuration of the neural network, the parallel computing configuration being used to characterize the computing and storage resources allocated to the neural network to be trained; and determining a recomputation strategy based on the parallel computing configuration.

[0143] In one embodiment, when the processor executes the computer program, it further performs the following steps: allocating training pipelines to the neural network and determining the storage resources and training time of each training pipeline according to the parallel computing configuration; and determining a recomputation strategy based on the storage resources and training time of each training pipeline.

[0144] In one embodiment, when the processor executes the computer program, it also performs the following steps: determining a training time determination function based on the parallel computing configuration; using each computing power resource and storage resource as constraints, solving for the minimum value of the training time determination function to obtain the storage resources and training time of each training pipeline segment.

[0145] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each training pipeline segment, it compares the storage resources with the local storage resources corresponding to the training pipeline segment; if the storage resources are greater than the local storage resources, it determines the training pipeline segment as the sending pipeline segment; if the storage resources are less than or equal to the local storage resources, it determines the training pipeline segment as the receiving pipeline segment.

[0146] In one embodiment, when the processor executes the computer program, it further performs the following steps: summing the storage resources corresponding to each transmitting pipeline segment to obtain a first candidate storage resource for each transmitting pipeline segment; summing the storage resources corresponding to each receiving pipeline segment to obtain a second candidate storage resource for each receiving pipeline segment; if the difference between the first candidate storage resource and the second candidate storage resource is greater than a difference threshold, then determining the communication parameters of each transmitting pipeline segment or each receiving pipeline segment, and determining each first storage resource and each second storage resource based on each communication parameter, the first candidate storage resource, and the second candidate storage resource, wherein the communication parameters are determined based on the shortest communication distance and the parallel communication load.

[0147] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the first storage resource is greater than the second storage resource, then the transmitting pipeline segment with the smallest communication parameter among the transmitting pipeline segments is updated to the receiving pipeline segment, thereby obtaining each first storage resource and each second storage resource; if the second storage resource is greater than the first storage resource, then the receiving pipeline segment with the smallest communication parameter among the receiving pipeline segments is updated to the transmitting pipeline segment, thereby obtaining each first storage resource and each second storage resource.

[0148] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: obtaining a recomputation strategy for a neural network to be trained, the recomputation strategy being used to characterize the storage resources and training time of each training pipeline segment, each training pipeline segment being used to perform parallel computation on the neural network; identifying the sending pipeline segment and the receiving pipeline segment in each training pipeline segment according to the recomputation strategy, and determining the first storage resource of each sending pipeline segment and the second storage resource of each receiving pipeline segment; training the neural network according to the first storage resource through each sending pipeline segment and according to the second storage resource through each receiving pipeline segment to obtain a trained target neural network, the target neural network being used for natural language processing.

[0149] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a parallel computing configuration of the neural network, the parallel computing configuration being used to characterize the computing and storage resources allocated to the neural network to be trained; and determining a recomputation strategy based on the parallel computing configuration.

[0150] In one embodiment, when the processor executes the computer program, it further performs the following steps: allocating training pipelines to the neural network and determining the storage resources and training time of each training pipeline according to the parallel computing configuration; and determining a recomputation strategy based on the storage resources and training time of each training pipeline.

[0151] In one embodiment, when the processor executes the computer program, it also performs the following steps: determining a training time determination function based on the parallel computing configuration; using each computing power resource and storage resource as constraints, solving for the minimum value of the training time determination function to obtain the storage resources and training time of each training pipeline segment.

[0152] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each training pipeline segment, it compares the storage resources with the local storage resources corresponding to the training pipeline segment; if the storage resources are greater than the local storage resources, it determines the training pipeline segment as the sending pipeline segment; if the storage resources are less than or equal to the local storage resources, it determines the training pipeline segment as the receiving pipeline segment.

[0153] In one embodiment, when the processor executes the computer program, it further performs the following steps: summing the storage resources corresponding to each transmitting pipeline segment to obtain a first candidate storage resource for each transmitting pipeline segment; summing the storage resources corresponding to each receiving pipeline segment to obtain a second candidate storage resource for each receiving pipeline segment; if the difference between the first candidate storage resource and the second candidate storage resource is greater than a difference threshold, then determining the communication parameters of each transmitting pipeline segment or each receiving pipeline segment, and determining each first storage resource and each second storage resource based on each communication parameter, the first candidate storage resource, and the second candidate storage resource, wherein the communication parameters are determined based on the shortest communication distance and the parallel communication load.

[0154] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the first storage resource is greater than the second storage resource, then the transmitting pipeline segment with the smallest communication parameter among the transmitting pipeline segments is updated to the receiving pipeline segment, thereby obtaining each first storage resource and each second storage resource; if the second storage resource is greater than the first storage resource, then the receiving pipeline segment with the smallest communication parameter among the receiving pipeline segments is updated to the transmitting pipeline segment, thereby obtaining each first storage resource and each second storage resource.

[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0156] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0157] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0158] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A neural network computation method based on spatiotemporal equalization, characterized in that, The method includes: Obtain the recomputation strategy of the neural network to be trained. The recomputation strategy is used to characterize the storage resources and training time of each training pipeline segment. Each training pipeline segment is used to perform parallel computation on the neural network. The transmitting pipeline and receiving pipeline in each training pipeline are identified according to the recomputation strategy, and the first storage resource of each transmitting pipeline and the second storage resource of each receiving pipeline are determined. The target neural network is trained by each of the transmitting pipelines based on the first storage resource and by each of the receiving pipelines based on the second storage resource. The target neural network is used for natural language processing.

2. The method according to claim 1, characterized in that, The recomputation strategy for obtaining the neural network to be trained includes: Obtain the parallel computing configuration of the neural network, which is used to characterize the computing and storage resources allocated to the neural network to be trained; The recomputation strategy is determined based on the parallel computing configuration.

3. The method according to claim 2, characterized in that, Determining the recomputation strategy based on the parallel computing configuration includes: Each training pipeline segment is allocated to the neural network, and the storage resources and training time of each training pipeline segment are determined according to the parallel computing configuration; The recomputation strategy is determined based on the storage resources and training time of each training pipeline segment.

4. The method according to claim 3, characterized in that, The step of determining the storage resources and training time for each training pipeline segment based on the parallel computing configuration includes: Based on the parallel computing configuration, a training time determination function is determined; Using the computing resources and storage resources as constraints, the minimum value of the training time determination function is solved to obtain the storage resources and training time of each training pipeline segment.

5. The method according to any one of claims 1-4, characterized in that, The step of identifying the transmitting pipeline segment and the receiving pipeline segment in each pipeline segment according to the recomputation strategy includes: For each training pipeline segment, a comparison is made between the storage resources and the local storage resources corresponding to the training pipeline segment; If the storage resources are greater than the local storage resources, the training pipeline segment is determined to be the sending pipeline segment; If the storage resource is less than or equal to the local storage resource, the training pipeline segment is determined to be the receiving pipeline segment.

6. The method according to claim 5, characterized in that, The determination of the first storage resources of each of the transmitting pipeline segments and the second storage resources of each of the receiving pipeline segments includes: The storage resources corresponding to each of the sending end pipelines are summed to obtain the first candidate storage resources for each of the sending end pipelines; The storage resources corresponding to each of the receiving end pipeline segments are summed to obtain the second candidate storage resources for each of the receiving end pipeline segments; If the difference between the first candidate storage resource and the second candidate storage resource is greater than the difference threshold, then the communication parameters of each of the transmitting end pipelines or each of the receiving end pipelines are determined. Based on the communication parameters, the first candidate storage resource and the second candidate storage resource, each of the first storage resources and each of the second storage resources are determined. The communication parameters are determined based on the shortest communication distance and the parallel communication load.

7. The method according to claim 6, characterized in that, The step of determining each first storage resource and each second storage resource based on each of the communication parameters, the first candidate storage resource, and the second candidate storage resource includes any one of the following: If the first storage resource is greater than the second storage resource, then the sending pipeline segment with the smallest communication parameters in each sending pipeline segment is updated to the receiving pipeline segment, thus obtaining each first storage resource and each second storage resource; If the second storage resource is greater than the first storage resource, then the receiver pipeline segment with the smallest communication parameter among the receiver pipeline segments is updated to the transmitter pipeline segment, thus obtaining each first storage resource and each second storage resource.

8. A neural network computing device based on spatiotemporal equalization, characterized in that, The device includes: The strategy acquisition module is used to acquire the recomputation strategy of the neural network to be trained. The recomputation strategy is used to characterize the storage resources and training time of each training pipeline segment. Each training pipeline segment is used to perform parallel computation on the neural network. The identification module is used to identify the transmitting pipeline segment and the receiving pipeline segment in each of the training pipeline segments according to the recomputation strategy, and to determine the first storage resource of each of the transmitting pipeline segments and the second storage resource of each of the receiving pipeline segments. The training module is used to train the neural network according to the first storage resource through each of the sending end pipelines and according to the second storage resource through each of the receiving end pipelines to obtain a trained target neural network, which is used for natural language processing.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images