Distributed training method and apparatus, device, and storage medium

Abstract

The application provides a distributed training method and device, equipment and a storage medium, and relates to the technical field of machine learning. The method is applied to a first computing device in a distributed training system, and the distributed training system comprises the first computing device and at least one other computing device. The distributed training method comprises the following steps: receiving input data, wherein the input data comprises a prediction tensor shard and a global label tensor; generating local statistical information corresponding to at least one sample respectively according to the prediction tensor shard and the global label tensor; sending first local statistical information to each other computing device, receiving second local statistical information from the other computing devices, and obtaining global statistical information; and generating a loss value corresponding to at least one sample respectively according to the global statistical information. The method improves the efficiency of distributed training.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a distributed training method, apparatus, device, and storage medium. Background Technology

[0002] In the field of machine learning technology, when tensors are used for training in Large Language Models (LLM), the vocabulary dimension of the global prediction tensor output by the model is split across different computing devices, meaning that each computing device only holds a local fragment of the prediction tensor.

[0003] In related technologies, any computing device typically needs to perform multiple rounds of computation and aggregation communication operations in sequence to calculate the loss value of the local prediction tensor fragment, and generate and store intermediate tensors during the execution process.

[0004] However, due to the large vocabulary dimension of the intermediate tensor, as the model size and / or data size increase, the training task will be limited by insufficient storage space, which will lead to a decrease in the efficiency of distributed training. Summary of the Invention

[0005] This application provides a distributed training method, apparatus, device, and storage medium. The technical solutions provided by this application are as follows.

[0006] According to one aspect of the embodiments of this application, a distributed training method is provided, applied to a first computing device in a distributed training system, the distributed training system including the first computing device and at least one other computing device, the method comprising: Receive input data, which includes prediction tensor slices and a global label tensor, wherein the global label tensor includes target labels corresponding to multiple samples respectively; Based on the prediction tensor slices and the global label tensor, at least one sample's local statistical information is generated; wherein, the sample's local statistical information includes: local maximum, local exponent sum, and the prediction score corresponding to the sample's target label; The system sends first local statistical information to each of the other computing devices and receives second local statistical information from the other computing devices to obtain global statistical information; wherein, the first local statistical information includes local statistical information corresponding to the at least one sample generated by the first computing device, and the second local statistical information includes local statistical information corresponding to the at least one sample generated by the other computing devices. Based on the global statistical information, a loss value is generated for each of the at least one sample.

[0007] According to one aspect of the embodiments of this application, a distributed training apparatus is provided, which is applied to a first computing device in a distributed training system, the distributed training system including the first computing device and at least one other computing device, the apparatus comprising: The receiving module is used to receive input data, which includes prediction tensor slices and global label tensors, wherein the global label tensor includes target labels corresponding to multiple samples respectively; The local statistics module is used to generate local statistical information corresponding to at least one sample based on the prediction tensor slices and the global label tensor; wherein, the local statistical information corresponding to the sample includes: local maximum value, local index sum, and the prediction score corresponding to the target label of the sample; A global statistics module is used to send first local statistical information to each of the other computing devices and receive second local statistical information from the other computing devices to obtain global statistical information; wherein, the first local statistical information includes local statistical information corresponding to the at least one sample generated by the first computing device, and the second local statistical information includes local statistical information corresponding to the at least one sample generated by the other computing devices. The loss calculation module is used to generate loss values ​​corresponding to the at least one sample based on the global statistical information.

[0008] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described distributed training method.

[0009] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described distributed training method.

[0010] According to one aspect of the embodiments of this application, a chip is provided, the chip including programmable logic circuits and / or program instructions, which, when the chip is running, are used to implement the above-described distributed training method.

[0011] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program, the computer program being loaded and executed by a processor to implement the above-described distributed training method.

[0012] The technical solution provided in this application can bring the following beneficial effects: After receiving input data containing prediction tensor fragments and a global label tensor, this approach generates local statistics including local maxima, local exponents, and the predicted score corresponding to the target label of each sample. Global statistics are obtained by exchanging these local statistics across computing devices, and then the loss value for each sample is generated based on these global statistics. This technical solution extracts prediction tensor fragments, whose data volume is related to the size of the local vocabulary, into local statistics that are only related to the number of samples. Each sample corresponds to a set of statistics consisting of local maxima, local exponents, and the predicted score corresponding to the target label, and its data size is much smaller than that of the prediction tensor fragments. Furthermore, since subsequent cross-device communication and loss calculation are based on the aforementioned local statistics, there is no need to generate and store intermediate tensors of the same data size as the global prediction tensor, thus avoiding excessive storage space consumption caused by storing intermediate tensors. As the model size and / or data size increases, the technical solution provided in this application can still maintain a low storage space footprint, and the training task will not be limited by insufficient storage space, thereby effectively improving the efficiency of distributed training. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a distributed training method provided in one possible implementation of this application; Figure 2 This is a flowchart of a method for determining local maxima and local exponent sums provided in one possible implementation of this application; Figure 3 This is a schematic diagram of a distributed training process provided in one possible implementation of this application; Figure 4 This is a structural block diagram of a distributed training device provided in one possible implementation of this application; Figure 5 This is a structural block diagram of a computer device provided in one possible implementation of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0016] Before introducing the technical solution proposed in this application, the relevant technical background and related technologies will be briefly described below.

[0017] In the field of machine learning technology, when training large language models using tensor parallelism, the vocabulary dimension of the global prediction tensor of the model's output layer is split across multiple computing devices (e.g., graphics processing units, GPUs), meaning each computing device only holds a local prediction tensor slice corresponding to that device. When calculating the loss value based on the cross-entropy loss function, related techniques typically require multiple rounds of computation and ensemble communication operations. For example, a distributed loss calculation method implemented in the PyTorch deep learning framework is described, which includes the following steps: Step 1. Each computing device first calculates the maximum value of the prediction tensor fragment it holds, and obtains the maximum value of the global prediction tensor through the first All-Reduce communication operation.

[0018] Step 2. Each computing device starts its computing kernel, obtains the maximum value of the global prediction tensor as the global maximum value, performs subtraction operation on the prediction tensor slices using the above global maximum value to obtain the first intermediate tensor (denoted as shiftedlogits), and further performs exponential operation on the first intermediate tensor to obtain the second intermediate tensor (denoted as shiftedexp).

[0019] Step 3. Each computing device starts the summation kernel, sums the second intermediate tensor shifted exp on the specified dimension to obtain the local exponential sum, and obtains the exponential sum of the global prediction tensor (denoted as shifted exp sum) through the second All-Reduce communication operation.

[0020] Step 4. Each computing device starts the logarithmic operation kernel, performs logarithmic operation on the above shifted exp sum, and subtracts the logarithmic operation result from the above first intermediate tensor shifted logits to obtain the third intermediate tensor (denoted as log_softmax).

[0021] Step 5. Each computing device extracts the corresponding loss value from the third intermediate tensor log_softmax based on the target label. For variable-length sequence inputs, this step also requires launching an additional kernel to handle invalid padding positions in order to filter out valid target labels. Finally, through the third All-Reduce communication operation, the loss values ​​of each computing device are aggregated to obtain the global loss value.

[0022] It is worth noting that the relevant technologies have the following technical problems: In the above calculation process, due to the data dependencies between each calculation step, the subsequent calculation must wait for the output of the previous step as input. Specifically: the exponential operation in step 2 depends on the maximum value of the global prediction tensor calculated in step 1; the summation operation in step 3 depends on the second intermediate tensor output in step 2; the logarithmic operation in step 4 depends on the global exponential sum output in step 3 and the first intermediate tensor output in step 2; and the loss value extraction in step 5 depends on the third intermediate tensor output in step 4. Therefore, to satisfy the above data dependencies, the intermediate tensors generated after each calculation must be completely stored in storage space (e.g., video memory) for subsequent steps to read. Since the vocabulary dimension of the intermediate tensors is the same as that of the global prediction tensor, as the model size (i.e., vocabulary dimension) and / or data size (i.e., batch size) increases, the storage space occupied by the intermediate tensors will also increase, leading to insufficient storage space during training tasks, and in severe cases, it may cause problems such as video memory overflow (OOM). In summary, the technical solutions provided by the relevant technologies limit the size of trainable models and the scale of data, which in turn leads to a decrease in training efficiency.

[0023] To address the aforementioned issues, this application provides a distributed training method.

[0024] It is worth noting that the distributed training method provided in this application is applicable to distributed training systems, which include multiple interconnected computing devices capable of data communication. For ease of description, one of the computing devices is referred to as the first computing device, and the remaining computing devices as other computing devices; however, this designation is merely illustrative and does not imply a hierarchy in function or structure. The distributed training system can be configured to employ tensor parallelism, data parallelism, pipelined parallelism, or any combination thereof to support efficient training of large-scale models (such as LLM) in a multi-device environment. The computing devices communicate collectively via a high-speed interconnect network or bus to synchronize and exchange model parameters, gradients, statistics, loss values, or intermediate results.

[0025] In one aspect of this embodiment, "computing device" broadly refers to a hardware or software execution entity capable of performing at least a portion of the steps in the distributed training method provided in this application, and its specific form is not limited. For example, a computing device may include: a single processor or processor core (e.g., a Central Processing Unit (CPU)); a GPU, Tensor Processing Unit (TPU), or other dedicated accelerators; a server node composed of multiple processors or accelerators; a process or thread running in a virtual machine environment; a cloud computing instance or an edge computing node. In short, a computing device can be a physical hardware unit or a logical execution unit. As long as it possesses the necessary computing power, memory resources, and communication interfaces, and can participate in local computation, ensemble communication, and loss value generation operations in distributed training, it falls within the scope of the computing device referred to in this application.

[0026] In one aspect of this embodiment, the distributed training method proposed in this application has flexibility in design granularity, and can be implemented on different levels of computing units according to actual deployment needs. Specifically, it may include: 1. Process-level granularity: Distributed training methods can be deployed across multiple independent processes, each distributed across the same or different physical devices, and data exchange is achieved through inter-process communication mechanisms (such as shared memory). This approach is suitable for distributed training environments with single-machine multi-process or multi-machine multi-process architectures.

[0027] 2. Device-level granularity: Distributed training methods can be deployed across multiple independent GPUs, TPUs, or other processors. Each computing device directly undertakes part of the model computation and communication tasks, and can utilize high-speed links between devices (such as PCIe (Peripheral Component Interconnect express)) for aggregated communication. This approach is suitable for large-scale model training that requires high-bandwidth, low-latency communication.

[0028] 3. Node-level granularity: Distributed training methods can be deployed on server nodes consisting of multiple processors or accelerators. Each node participates in distributed training as a computing device, and data synchronization between nodes is achieved through network interconnection.

[0029] It is worth noting that the above-mentioned implementation methods with different granularities can be nested or combined in logical structure. For example, process-level parallelism can be used within a node, while device-level or node-level parallelism can be used between nodes. The scope of protection of this application covers all granularity selections and architectural combinations that can implement the distributed training method provided in this application, and is not limited to any specific hardware topology or communication framework.

[0030] In one aspect of this embodiment, a collaborative working mechanism exists between the computing devices in the distributed training system. Each computing device can act as both a data provider and a data receiver during the training process, and obtains globally available relevant information or data after collective communication operations. For ease of description of the method flow, this application refers to one of the computing devices as the first computing device to exemplify the execution process. In fact, all computing devices are functionally equivalent, and any computing device can serve as the first computing device to execute all or part of the steps of the distributed training method proposed in this application, while other computing devices can be replaced by the first computing device without affecting the implementation of the technical solution. Therefore, the naming of the first computing device in this application does not constitute a limitation on the scope of protection, and any computing device that conforms to the definition of this application falls within the scope of protection.

[0031] In summary, the distributed training method proposed in this application does not depend on specific hardware implementations, communication libraries, or topologies. Those skilled in the art can flexibly deploy it at the process level, device level, node level, or higher granularity according to actual needs, and the roles of each computing device can be interchanged. All of the above implementation methods do not depart from the core concept and protection scope of this application.

[0032] Based on the above definitions and architecture descriptions of distributed training systems and computing devices, this application provides a possible implementation of a distributed training method. For example, please refer to... Figure 1 , Figure 1 This is a flowchart of a distributed training method provided in one possible implementation of this application. The distributed training method described above may include at least one of the following steps 110 to 140.

[0033] Step 110: Receive input data, which includes prediction tensor slices and global label tensors. The global label tensor includes target labels corresponding to multiple samples.

[0034] In one aspect of this embodiment, during distributed training, especially in scenarios where a tensor parallel strategy is used to train an LLM, the global prediction tensor of the model's output layer is divided into multiple slices according to a specified dimension (e.g., vocabulary dimension), and stored on different computing devices. Each computing device only holds its corresponding prediction tensor slice. In the above steps, the input data received by the first computing device includes its held prediction tensor slices and the global label tensor.

[0035] In one aspect of this embodiment, a prediction tensor slice refers to a subset or local fragment of the global prediction tensor along a specific dimension. The global prediction tensor is the output tensor generated by the model's output layer during the model's forward propagation, representing the model's original prediction preferences or scores for each preset category, word, or prediction position. Since the storage capacity of a single computing device is limited and cannot accommodate a complete global prediction tensor (especially when the vocabulary or category dimensions are large), a tensor parallel strategy can be used to divide the global prediction tensor into multiple non-overlapping slices along a specified segmentation dimension (usually the category or vocabulary dimension), and store them separately on various computing devices in the distributed training system. Each computing device holds only one slice corresponding to it, i.e., the prediction tensor slice proposed in this application.

[0036] For example, prediction tensor slices can have the following characteristics: First, in terms of dimensions, prediction tensor slices are consistent with the global prediction tensor in all dimensions except the splitting dimension (e.g., batch size, sequence length). For example, suppose the dimensions of the global prediction tensor are represented as (N, C_global), where N represents the sample dimension (which may include batch size and / or sequence length), and C_global represents the global category or vocabulary size. After splitting along the C_global dimension, the dimension of the prediction tensor slice held by any computing device is (N, C_local), where C_local is the local category or vocabulary size handled by any computing device, and the sum of C_local values ​​of all computing devices equals C_global. Second, in terms of numerical meaning, each element value in the prediction tensor slice can be regarded as the model's prediction score for the corresponding category or vocabulary. For example, the prediction score can be the raw output value without normalization, or it can be a value after some transformation (such as logarithmic transformation, scaling, or other linear / nonlinear changes), as long as it can reflect the model's preference for each possible output. Finally, from a storage perspective, prediction tensor fragments can be stored as tensors in the local memory of the computing device (e.g., GPU memory). It should be noted that this application does not strictly limit the specific details regarding the number of dimensions of the prediction tensor (e.g., two-dimensional, three-dimensional, or higher-dimensional), the segmentation method (e.g., segmentation by category dimension, vocabulary dimension, sequence dimension, or any combination thereof), or the storage method. As long as each computing device holds a global prediction tensor fragment, it falls within the protection scope of the prediction tensor fragments proposed in this application.

[0037] In one aspect of this embodiment, the global label tensor is a data structure used to store the true label information of samples, that is, the global label tensor includes target labels corresponding to multiple samples respectively. A sample is the basic data unit constituting one training iteration. In natural language processing tasks, a sample can be a token, a sentence, or a document; in image classification tasks, a sample can be an image; in speech recognition tasks, a sample can be an audio frame or a segment of speech. This application does not limit the specific granularity and form of the sample; it can be any basic unit that the model needs to predict. For each sample, the global label tensor includes a corresponding target label. The target label is information representing the true category to which the sample belongs or the correct prediction result, usually existing in the form of an index, code, or identifier. For example, in classification tasks, the target label can be a category index value; in sequence generation tasks, the target label can be the index of the next token; in multi-label classification tasks, a sample may correspond to multiple target labels. The first dimension of the global label tensor (which can be called the sample dimension) should correspond to the sample dimension of the prediction tensor slice to ensure that the prediction information of each sample is accurately aligned with its target label. For example, if the prediction tensor slice dimension is (N, C_local), then the global label tensor can be a one-dimensional tensor of dimension (N,), where each element is the target label of the corresponding sample. In other possible implementations, depending on the task complexity, the global label tensor can also have higher dimensions, such as (N, K) to represent a multi-label scenario where each sample has K target labels, or (N, S) to represent a sequence labeling scenario where each sample contains target labels at S sequence positions. It is worth noting that, regardless of the specific form used, the core function of the global label tensor is to provide accurate and aligned true label information (i.e., the aforementioned target labels) for each sample, which is used to subsequently calculate the difference between the prediction result and the true result, i.e., the loss value.

[0038] In one aspect of this embodiment, the prediction tensor slices and the global label tensor together constitute the input foundation of the distributed training method. A precise dimensional alignment relationship exists between them. For example, the first dimension (sample dimension N) in the prediction tensor slice corresponds one-to-one with the sample dimension in the global label tensor. This alignment relationship ensures that for any index i (0 ≤ i < N), the predicted score of the i-th sample across all local categories can be uniquely obtained from the prediction tensor slice, and the target label of the i-th sample can be obtained from the global label tensor. It is precisely based on this alignment relationship that subsequent steps can accurately associate the predicted information of each sample with the true label, thereby calculating the local statistics and loss value for each sample.

[0039] In one aspect of this embodiment, during one iteration of forward propagation in the distributed training system, the first computing device first receives input data. Specifically, the implementation of the above process may include: the first computing device reading pre-stored prediction tensor fragments generated by the model's forward computation from its local memory. Simultaneously, the first computing device obtains the global label tensor. It should be noted that the global label tensor in the distributed training system can typically exist in a broadcast or shared manner, meaning each computing device can hold the complete global label tensor. Because the storage overhead of the label data itself is relatively small compared to the model parameters, the first computing device can directly obtain the complete global label tensor from local storage or through a data loader. After completing the above reading operation, the first computing device has completed the "receiving input data" process proposed in step 110.

[0040] For example, in a specific embodiment based on the PyTorch framework, the distributed training method proposed in this application can be encapsulated as a custom autograd function. This function can serve as an interface directly invoked by the user to receive input data. Specifically, the input data may include: 1. Prediction Tensor Sharding: In tensor parallel mode, the prediction tensor shard is a distributed tensor (DTensor) with shape (N, C_local), where N is the product of the batch size and the sequence length (i.e., the number of samples), and C_local is the size of the local vocabulary held by the first computing device. This prediction tensor shard can be stored in the GPU memory of the first computing device, including the prediction scores corresponding to the local vocabulary handled by the first computing device.

[0041] 2. Global Label Tensor: The global label tensor is a one-dimensional tensor of shape (N,) stored in the memory or video memory of the first computing device. Here, (N,) indicates that the global label tensor has only one dimension, with a length of N, corresponding to the sample dimension. For example, each element in the global label tensor can be an integer index, ranging from 0 to C_global-1, indicating the position of the target label of the corresponding sample in the global vocabulary.

[0042] On the other hand, in addition to the aforementioned prediction tensor slices and global label tensors, the input data can also include, but is not limited to, aggregation modes and communication groups. The aggregation mode specifies the aggregation method for the loss values, such as summation or averaging. The communication group identifies the set of computing devices participating in the current ensemble communication, i.e., the communication group formed by the various computing devices participating in distributed training. Through the above interface definition, users only need to input the prediction tensor slices and global label tensors output by the model, and configure the corresponding aggregation mode and communication group to start the subsequent distributed training process, without needing to worry about the underlying distributed splitting details.

[0043] It is worth noting that the implementation based on the PyTorch framework in the above embodiments is merely a specific application example of this application and does not constitute a limitation on the scope of protection of this application. In practical applications, those skilled in the art can make corresponding adjustments and modifications to the organization, storage method, and acquisition method of input data according to different deep learning frameworks (such as TensorFlow, PaddlePaddle, etc.), different hardware platforms (such as CPU, GPU, TPU, etc.), and different parallel strategies (such as data parallelism, pipelined parallelism, or a hybrid parallelism). For example, the global label tensor can also be stored in the form of fragments, as long as the collective communication can ensure that each computing device obtains the label information required for its computation. Any technical solution based on the core concept of this application, which receives input data including prediction tensor fragments and label information for aligning samples to start subsequent distributed training, falls within the scope of protection of this application.

[0044] Step 120: Based on the prediction tensor fragmentation and the global label tensor, generate local statistical information corresponding to at least one sample. The local statistical information for each sample includes: local maxima, local exponents, and the predicted score corresponding to the sample's target label.

[0045] In one aspect of this embodiment, after receiving the input data, the first computing device needs to generate corresponding local statistical information for each sample based on the prediction tensor slices and the global label tensor. Local statistical information is an intermediate data format obtained by extracting and compressing the local prediction information contained in the prediction tensor slices. Its core function is to carry the numerical features necessary for subsequent calculations with a small amount of data, thereby avoiding the generation and storage of large-scale intermediate tensors during the calculation process. In another aspect of this embodiment, the local statistical information is generated independently for each sample. For any given sample, its corresponding local statistical information may include the following three types: local maximum, local exponential sum, and the prediction score corresponding to the target label of the sample.

[0046] In one aspect of this embodiment, a local maximum refers to the maximum value among all predicted scores for the current sample within a local area covered by the prediction tensor slice held by the first computing device. Since the prediction tensor slice only contains a portion of the global prediction tensor (e.g., a local slice of the vocabulary dimension), the aforementioned maximum value only reflects the maximum prediction preference within the local category set managed by the first computing device, rather than the maximum value globally. The role of the local maximum is to provide a benchmark for subsequent numerical stabilization calculations. For example, when calculating local exponential sums, the participation of the local maximum in the above calculation process can effectively avoid numerical overflow.

[0047] In one aspect of this embodiment, the local exponential sum refers to the summation of the results of exponential operations performed on all predicted scores corresponding to the current sample within the local area covered by the predicted tensor slice held by the first computing device (e.g., exponential operation with the natural constant e as the base). It should be noted that, in the specific calculation process, to ensure numerical stability, the exponential operation is usually performed after subtracting the local maximum value (or the subsequently obtained global maximum value), i.e., the calculation is in the form of exp(predicted score - maximum value). It is worth noting that this application does not strictly limit the specific implementation of the exponential operation (such as the choice of base, whether to combine it with the maximum value for offsetting, etc.), as long as the result can be used for subsequent loss value calculation.

[0048] In one aspect of this embodiment, the predicted score corresponding to the target label of a sample refers to the predicted score extracted from the prediction tensor slice held by the first computing device for the current sample, corresponding to the target label of that sample. This predicted score is one of the core input values ​​required for calculating the final loss value, directly reflecting the model's prediction preference for the true category. It should be noted that since the prediction tensor slice only covers local categories within the global category, the predicted score corresponding to the target label may or may not exist on the first computing device (i.e., the target label of the sample belongs to the category range handled by other computing devices). If the target label belongs to the slice range handled by the first computing device, the predicted score is the specific value at the corresponding position in the prediction tensor slice; if the target label does not belong to the slice range handled by the first computing device, the predicted score cannot be directly obtained from the local slice. In this case, a preset value (e.g., negative infinity, zero, or a very small value) is needed to place or mark the position, indicating that the target label of the sample is not local. The above design ensures that, regardless of the target label, each sample's local statistics contain a predicted score corresponding to the target label, thus guaranteeing the data structure integrity and consistency of subsequent global statistics.

[0049] The aforementioned local statistics (including local maxima, local indices, and the predicted scores corresponding to the target labels of the samples) serve as an information bridge between the prediction tensor slices and the loss value calculation. By generating and transmitting local statistics, computing devices can collaboratively calculate global statistics and the loss value for each sample in subsequent steps without exchanging complete prediction tensor slices and / or intermediate tensors.

[0050] In one aspect of this embodiment, for each sample, the first computing device can access a portion of the data corresponding to that sample in the prediction tensor slice, wherein the portion of data includes multiple prediction scores for that sample within a local range. Based on these multiple prediction scores, the first computing device can determine the sample's local maximum, local exponent sum, and the prediction score corresponding to the target label by performing a series of computational operations. Specifically, for each sample, the first computing device can perform the following operations: 1. Identifying Local Maximums. The first computing device can identify the local maximum by analyzing the numerical relationships among multiple predicted scores corresponding to the sample. For example, the local maximum can be selected from multiple predicted scores through comparison, sorting, or traversal.

[0051] 2. Determine the local exponential sum. The first computing device can perform exponential operations on multiple predicted scores corresponding to the sample, and sum the results of all exponential operations to obtain the local exponential sum of the sample. To ensure numerical stability, the exponential operation can also be corrected by incorporating local maxima, for example, by calculating the difference between each predicted score and the local maximum before performing the exponential operation.

[0052] 3. Obtain the prediction score corresponding to the target label. The first computing device can determine whether the target label falls within the range covered by the local prediction tensor slice based on the index information of the target label. If so, the prediction score corresponding to the target label index is extracted from the multiple prediction scores corresponding to the sample; if not, the prediction score corresponding to the target label of the sample is set to a preset value, such as a specific value indicating "not present" or "invalid", such as negative infinity or zero.

[0053] It should be noted that the above operations do not necessarily have to be performed in a strict order. They can be performed sequentially, in parallel, or combined in any interleaved manner. For example, updating local maxima, accumulating local exponent sums, and extracting predicted scores corresponding to target labels can be completed simultaneously during a single traversal. This step does not impose restrictions on the specific execution order and implementation method, as long as it ultimately generates local statistical information containing the above three types of information for each sample. Furthermore, for certain special samples, such as filled positions or invalid samples, the complete calculation process described above can be skipped, and their local statistical information can be directly set to preset default values ​​to avoid invalid calculations.

[0054] In one aspect of this embodiment, the core of step 120 lies in the concept of generating local statistical information containing local maxima, local exponential sums, and the predicted score corresponding to the target label based on the prediction tensor fragmentation and the global label tensor, rather than a specific parallel scheduling strategy, numerical calculation method, or programming model. In practical applications, those skilled in the art can modify and adjust the generation method of local statistical information according to different hardware architectures, different programming frameworks, and different performance optimization requirements. For example, thread bundles can be used instead of threads as processing units to improve data reuse and parallel efficiency; different numerical stabilization strategies can be used, such as first calculating the local maxima and then uniformly performing exponential operations to obtain the local exponential sum; vectorized loading instructions can be used to read multiple predicted scores at once to improve the utilization of video memory bandwidth; for cases where the target label is not local, in addition to setting it to negative infinity, it can also be set to a specific flag bit or sentinel value, as long as it can be correctly identified and processed in subsequent steps; the three types of operations can be completely decoupled and executed step by step; or they can be deeply integrated to complete the extraction of all information in one calculation.

[0055] It is worth noting that any technical solution based on the core concept proposed in this application, which extracts local statistical information including local maxima, local exponents, and the predicted scores corresponding to the target labels from multiple samples in the prediction tensor slices to support subsequent loss value calculation, falls within the protection scope of this application, regardless of its specific implementation details.

[0056] Step 130: Send first local statistical information to each of the other computing devices and receive second local statistical information from the other computing devices to obtain global statistical information. The first local statistical information includes local statistical information corresponding to at least one sample generated by the first computing device, and the second local statistical information includes local statistical information corresponding to at least one sample generated by the other computing devices.

[0057] After generating local statistical information for each sample, the first computing device can collaborate with other computing devices to aggregate the local statistical information scattered across various computing devices to obtain global statistical information for subsequent loss calculation.

[0058] In one aspect of this embodiment, the first local statistical information refers to the local statistical information corresponding to at least one sample generated and held by the first computing device. The second local statistical information refers to the local statistical information corresponding to at least one sample generated and held by various other computing devices in the distributed training system. It should be emphasized that, for any sample, the first local statistical information is the statistical information generated based on the prediction tensor slice held by the first computing device, while the second local statistical information is the statistical information generated based on the prediction tensor slice held by other computing devices.

[0059] It should be noted that since the first local statistical information is generated based on the prediction tensor fragments held by the first computing device, while the second local statistical information is generated based on the prediction tensor fragments held by other computing devices, and the prediction tensor fragments held by different computing devices correspond to different fragment ranges in the global prediction tensor, for the same sample, the local maximum and local exponent sum in the first local statistical information usually differ from those in the second local statistical information. Furthermore, there can be multiple other computing devices, each corresponding to a set of second local statistical information. Since the prediction tensor fragments held by different other computing devices correspond to different fragment ranges in the global prediction tensor, for the same sample, the second local statistical information provided by different other computing devices often also differs. In summary, precisely because the local statistical information generated by each computing device only reflects the statistical characteristics within its local data range, no single computing device can obtain the complete information required to calculate the loss value independently; therefore, aggregated communication operations are needed for information exchange and aggregation.

[0060] In one aspect of this embodiment, the first computing device sends first local statistical information to each of the other computing devices and receives second local statistical information from the other computing devices; this process constitutes a collective communication operation. After completing the above collective communication operation, the first computing device integrates the first local statistical information with the received second local statistical information to obtain global statistical information.

[0061] In one aspect of this embodiment, ensemble communication operation refers to the mechanism by which multiple computing devices in a distributed training system exchange and synchronize data according to a specific communication mode. Exemplarily, common ensemble communication operations include, but are not limited to, broadcasting, scattering, collection, full collection, and reduction. In this embodiment, the specific type of ensemble communication operation can be selected according to actual needs. For example, if each computing device needs to obtain local statistical information from all other computing devices, a full collection operation can be used. An all-gather operation means that each computing device sends its local data to all other devices while simultaneously receiving data from all other devices, ultimately giving each device a complete data set from all devices. Compared to multiple point-to-point communications, ensemble communication operations can utilize network bandwidth more efficiently and reduce communication latency. The core purpose of the ensemble communication operation performed in this step is to exchange and integrate the local statistical information generated by each computing device for the same batch of samples, enabling each computing device to obtain a complete statistical information view of all samples across all devices, i.e., global statistical information.

[0062] In one aspect of this embodiment, the global statistics are the result of performing ensemble communication. For each sample in the distributed training system, the global statistics integrate local statistics about that sample from all computing devices. Specifically, for any given sample, its global statistics include the local maximum, local exponent sum, and predicted score corresponding to the target label from computing device 1; the local maximum, local exponent sum, and predicted score corresponding to the target label from computing device 2; and so on, until all participating computing devices are covered. In summary, after completing ensemble communication, each computing device possesses a complete statistical triplet (i.e., global statistics) for each sample across all devices, preparing for subsequent calculations of the global maximum, global exponent sum, and the final loss value.

[0063] In one aspect of this embodiment, the input to step 130 is the local statistical information generated in step 120. If step 120 were not used to calculate and extract the predicted tensor pieces, and instead the original predicted tensor pieces were directly subjected to aggregate communication, it would result in significant communication overhead, reducing the efficiency of distributed training. It is precisely because step 120 compresses large-scale tensor data into smaller-scale local statistical information that the aggregate communication operation in step 130 becomes lightweight and efficient. In another aspect of this embodiment, the output of step 130 is global statistical information, which is the necessary input for loss calculation in step 140. If each computing device only possesses local statistical information from its own device and cannot know the statistical characteristics of other devices (e.g., local maximum values ​​on other devices), it is impossible to calculate the correct global maximum value and global exponential sum, and thus, the loss value cannot be accurately calculated. Therefore, step 130 ensures that subsequent loss calculations are based on complete global information, guaranteeing the correctness of the calculation results.

[0064] In one aspect of this embodiment, the implementation process of step 130 can be summarized as follows: the first computing device sends its locally generated first local statistical information for each sample to all other computing devices in the distributed training system via a set communication operation; simultaneously, the first computing device receives second local statistical information generated by each of the other computing devices; after completing the sending and receiving of all data, the first computing device merges its local first local statistical information with the received second local statistical information from the other devices to form global statistical information for all samples. Specifically, the above process may include, but is not limited to, the following stages: 1. Preparation Phase: The first computing device organizes the first local statistical information generated in step 120 into a format suitable for transmission. Since the first local statistical information is generated independently for each sample, its data volume is only related to the number of samples N, and not to the vocabulary size C_global. For example, for N samples, each sample corresponds to a triplet containing three values, so the total amount of data that the first computing device needs to send is N×3 values. These values ​​can usually be directly represented in floating-point form, resulting in a relatively small data volume.

[0065] 2. Initiation Phase: The first computing device invokes the ensemble communication interface provided by the distributed training framework to initiate an all-gather operation. When initiating the call, the communication group needs to be specified, which is the set of computing devices participating in this communication. The communication group includes the first computing device itself and all other computing devices.

[0066] 3. Execution Phase: At the underlying layer of the aggregated communication, each computing device sends its locally prepared local statistical information data to all other devices in the communication group. Simultaneously, each computing device continuously receives local statistical information from other devices. Throughout the communication process, data flows through the network, and ultimately each device receives local statistical information from all other devices.

[0067] 4. Integration Phase: After the first computing device has received all the data, its local memory contains local statistical information from all the computing devices. At this point, the first computing device organizes and integrates this information according to samples. Specifically, for each sample, the first computing device aggregates the local maximum values, local indices, and predicted scores corresponding to the target label from different devices to form the global statistical information for that sample. For example, the statistical information from different devices can be stored in a contiguous storage area according to device number or fragment order for access and processing in subsequent steps.

[0068] The following is a specific implementation based on the PyTorch framework. Step 130 is implemented by calling the distributed communication primitive.

[0069] In step 120, the first computing device generates local statistical information triples for each sample and stores these triples in a pre-allocated contiguous buffer. Assuming the number of samples is N, and each triple contains three floating-point numbers (local maximum, local exponential sum, and the predicted score corresponding to the target label), the size of this buffer is N × 3 floating-point numbers. Subsequently, in step 130, the first computing device can call the distributed communication interface provided by the PyTorch framework to perform an all-gather operation. Specifically, this can be implemented using the `torch.distributed.all_gather` function. For example, the function is called as follows: #local_stats: Local statistics on the current device, a tensor of shape (N, 3). #global_stats_list: A list used to receive statistics from all devices; the list length is the size of the communication group. Each element is a tensor of shape (N, 3) used to store local statistics of the corresponding computing device. #group: Communication group, which is the collection of computing devices participating in this communication. torch.distributed.all_gather(global_stats_list, local_stats, group=group) In the above call, local_stats is the local statistics tensor generated in step 120. After the all_gather function is executed, the local_stats data of each computing device will be collected into global_stats_list on all computing devices. Specifically, after execution, global_stats_list[0] stores the local statistics of computing device 0, global_stats_list[1] stores the local statistics of computing device 1, and so on.

[0070] At this point, the first computing device has obtained local statistical information from all computing devices through a single all-gather operation. Next, the first computing device can further integrate this information, for example, by concatenating all tensors in the global_stats_list along the first dimension to form a global statistical information tensor of shape (world_size×N, 3) or (N, world_size×3), so that subsequent steps can process it on a sample-by-sample basis.

[0071] It should be noted that the above all-gather operation only needs to be executed once to complete the exchange of all necessary information, significantly reducing the number of communications compared to schemes that require multiple gathering communication operations. Furthermore, since the amount of data transmitted is only N×3×world_size floating-point numbers, far less than the amount of data required to transmit a complete prediction tensor fragment (N×C_global floating-point numbers), the communication overhead is also greatly reduced.

[0072] It is important to emphasize that the specific implementation of the PyTorch framework and the all_gather primitive described above is merely an exemplary embodiment of this application and does not constitute a limitation on the scope of protection of this application. The core of this step lies in the overarching concept of "exchanging local statistical information to obtain global statistical information through set communication operations," rather than specific communication libraries, communication primitives, or data organization methods. In practical applications, those skilled in the art can make corresponding variations and adjustments to the implementation of set communication operations based on different distributed training frameworks, different hardware platforms, and different communication backends. For example: 1. All-reduce operations can be used instead of all-gather operations, if subsequent calculations can be performed directly on the reduction results without retaining the original statistical information of each device; 2. Asynchronous communication can be used to overlap communication and calculation to further hide communication latency; 3. Custom communication protocols or algorithms can be used to further calculate local statistical information before transmission to reduce communication volume; 4. For different network topologies (such as ring, tree, etc.), different communication algorithms can be selected to optimize communication efficiency.

[0073] Furthermore, the data organization of local statistical information can be flexibly adjusted. For example, the local maximum, local index, and predicted score corresponding to the target label of all samples can be stored as three independent tensors, and then aggregated for communication; alternatively, they can be stored interleaved as a single three-dimensional tensor. Regardless of the organization method, as long as the local statistical information scattered across various devices can ultimately be aggregated into a global view through aggregated communication, it falls within the protection scope of this application.

[0074] It is worth noting that any technical solution based on the core concept of this application that obtains global statistical information for subsequent loss calculation by exchanging local statistical information, regardless of its specific communication primitives, data formats or communication optimization methods, falls within the protection scope of this application.

[0075] Step 140: Based on global statistical information, generate loss values ​​for at least one sample.

[0076] In one aspect of this embodiment, after obtaining the global statistical information, the first computing device needs to generate a corresponding loss value for each sample based on the aforementioned global statistical information. The loss value is a quantitative indicator that measures the degree of difference between the model's predicted result and the true label, and it is the core basis for backpropagation and parameter updates during deep learning training. The global statistical information is a complete set of local statistical information for each sample from all computing devices, which is aggregated in step 130. For any sample, its global statistical information includes the local maximum value, local exponent sum from each computing device, and the predicted score corresponding to the target label of the sample (which may come from a certain computing device or may exist on all computing devices in the form of a preset value). This information together constitutes the complete data basis for calculating the loss value of the sample. The core task of step 140 is to calculate an accurate loss value for each sample based on the aforementioned global statistical information through a series of calculation operations. The loss value can be the loss component of a single sample, or it can be an intermediate result after preliminary aggregation (e.g., summation or averaging), depending on whether further cross-device loss aggregation is required subsequently.

[0077] In one aspect of this embodiment, the loss value is a scalar value used in machine learning to evaluate the difference between the model's prediction and the true label. In classification tasks, commonly used loss functions include cross-entropy loss and negative log-likelihood loss. The loss value generated in this application can be a calculation result corresponding to the aforementioned commonly used loss functions. For example, based on the cross-entropy loss function, the loss value of a sample can be expressed as the normalized negative logarithm of the predicted score corresponding to the target label of that sample. It is worth noting that this application does not strictly limit the specific mathematical form of the loss value, as long as it can be calculated based on global statistical information and can be used for subsequent backpropagation.

[0078] In one aspect of this embodiment, the implementation process of step 140 can be summarized as follows: the first computing device calculates the corresponding loss value for each sample based on the global statistical information obtained in step 130. Specifically, the above process includes, but is not limited to, the following stages: 1. Determining the Global Maximum and Global Exponential Sum: For each sample, the first computing device can determine the global maximum and global exponential sum of that sample across all computing devices from its global statistics. The global maximum is the largest of the local maximums across all devices, reflecting the highest predicted score for that sample across all local ranges. The global exponential sum is the corrected sum of the local exponential sums across all devices, reflecting the sum of the indexed predicted scores for that sample across all local ranges. Determining the global maximum can be achieved by reducing the local maximums from each computing device, for example, by taking the largest of all local maximums. Determining the global exponential sum requires correcting the local exponential sums of each device based on the global maximum before summing them. Specifically, since the local exponential sum of each device is calculated based on its own local maximum, and the global maximum may differ from the local maximum, it is necessary to first correct the local exponential sum of each device according to the difference between the global maximum and the local maximum, and then add all the corrected local exponential sums together to obtain the global exponential sum.

[0079] 2. Calculate the loss value for each sample: After obtaining the global maximum value and the global exponential sum, for each sample, the first computing device further calculates the loss value of the sample using the predicted score corresponding to the target label. The specific formula for calculating the loss value can be determined according to the type of loss function used. For example, if the cross-entropy loss function is used, the loss value of a sample can be calculated as follows: First, obtain the predicted score corresponding to the target label of the sample from the global statistics. It should be noted that this predicted score may come from a computing device (if the target label belongs to the segmentation range of that computing device), or it may be a preset value (if the target label is not within the segmentation range of any device). Then, subtract the global maximum value from the predicted score to ensure numerical stability, and then subtract the logarithm of the global exponential sum to finally obtain the loss value of the sample. Mathematically, this is equivalent to calculating the negative log-likelihood loss: loss = -(target label predicted score - global maximum value - log(global exponential sum)).

[0080] 3. Handling Special Cases: In actual training, some samples may require special handling. For example, in variable-length sequence training, samples at padding positions are invalid and should not be included in the loss calculation. For such samples, their loss value can be directly set to zero, or they can be excluded in subsequent steps using a masking mechanism. This step can combine the sample validity information to perform a complete loss calculation on valid samples, while assigning zero to invalid samples or skipping the calculation.

[0081] 4. Output and Storage of Loss Values: The calculated loss value for each sample can be stored in tensor form, for example, forming a one-dimensional tensor of shape (N, ), where each element corresponds to the loss value of a sample. This loss value tensor can be used as the output of forward propagation for use by upper-level callers, or it can be passed to the automatic differentiation framework as the starting point for backpropagation.

[0082] It is important to emphasize that the above description of how to generate loss values ​​based on global statistical information is a general implementation method for this step, rather than a limitation on its specific calculation details. The core of this step lies in the overarching concept of "generating sample loss values ​​based on global statistical information," rather than the specific form of the loss function, numerical stabilization strategy, or calculation order. In practical applications, those skilled in the art can modify and adjust the specific calculation method of the loss value according to different task types (such as classification, regression, sequence generation, etc.), different loss function definitions (such as cross-entropy, mean squared error, etc.), and different numerical precision requirements. For example: 1. Different loss functions can be used, only requiring corresponding adjustments to the calculation formula based on the global maximum value and the global exponential sum; 2. Global statistical information can be integrated into the global maximum value and the global exponential sum first, and then the loss value of all samples can be calculated uniformly; alternatively, samples can be processed one by one, calculating and outputting simultaneously; 3. For numerical stability, different offsets or scaling factors can be used, not limited to using the global maximum value for offset; 4. The calculation of the loss value can be combined with the pre-computation of the gradient, completing the generation of the loss value and gradient simultaneously in a single kernel call.

[0083] It is worth noting that any technical solution that generates a loss value for each sample by utilizing global statistical information based on the core concept of this application, regardless of its specific loss function form, calculation process or optimization method, falls within the protection scope of this application.

[0084] After receiving input data containing prediction tensor fragments and a global label tensor, this method generates local statistics including local maxima, local exponents, and the predicted score corresponding to the target label of the sample. Global statistics are obtained by exchanging these local statistics among computing devices, and then the loss value for each sample is generated based on the global statistics. This technical solution extracts prediction tensor fragments, whose data volume is related to the size of the local vocabulary, into local statistics that are only related to the number of samples. Each sample corresponds to a set of statistics consisting of local maxima, local exponents, and the predicted score corresponding to the target label, and its data size is much smaller than that of the prediction tensor fragments. Furthermore, since subsequent cross-device communication and loss calculation are based on the aforementioned local statistics, there is no need to generate and store intermediate tensors with the same data size as the global prediction tensor, thus avoiding excessive storage space consumption caused by storing intermediate tensors. As the model size and / or data size increases, the distributed training method provided in this application still maintains low storage space usage, and the training task is not limited by insufficient storage space, thereby effectively improving the efficiency of distributed training.

[0085] In some embodiments, based on the predicted tensor fragmentation and the global label tensor, local statistical information corresponding to at least one sample is generated, including: For each sample, read the multiple prediction scores corresponding to the sample contained in the prediction tensor slice, and the target label of the sample contained in the global label tensor.

[0086] Based on the multiple predicted scores corresponding to the sample, determine the local maximum and local exponent sum corresponding to the sample.

[0087] If the target label of a sample belongs to the segmentation range that the first computing device is responsible for, then the prediction score corresponding to the target label of the sample is obtained from the multiple prediction scores corresponding to the sample.

[0088] In one aspect of this embodiment, during the process of generating local statistical information based on the prediction tensor fragments and the global label tensor, a specific processing method for each sample is further determined. This processing method mainly includes two core aspects: first, determining the local maximum value and local exponent sum of the sample based on multiple prediction scores corresponding to the sample; second, obtaining the prediction score corresponding to the target label of the sample based on the attribution of the target label in the global label tensor. Specifically, for each sample, the first computing device needs to read the data corresponding to that sample from the prediction tensor fragments it holds. For example, assuming that the prediction tensor fragment is a two-dimensional data structure (N×C_local) organized according to the sample dimension and the local vocabulary dimension, for any sample index i, its corresponding vector in the prediction tensor fragment is a vector of length C_local, which contains all the prediction scores of that sample within the local vocabulary range managed by the current device. The reading operation can be loading the entire vector at once or accessing it element by element sequentially, depending on the needs of subsequent calculations and the characteristics of the hardware platform. After reading multiple predicted scores corresponding to a sample, the first computing device determines the local maximum and local exponential sum of the sample based on these predicted scores through numerical calculations. The local maximum is the highest value among these predicted scores, reflecting the model's strongest predictive tendency for the sample within the current local range. The local exponential sum is the accumulated result after performing exponential operations on these predicted scores, and it forms the basis for constructing the global normalized denominator. The process of determining these two values ​​can be implemented based on basic operations such as traversal, comparison, and accumulation, which are not specifically limited here.

[0089] In one aspect of this embodiment, for each sample, in addition to the local maximum and local exponent sum, it is also necessary to obtain the predicted score corresponding to the target label of that sample. The implementation of this operation depends on whether the index of the target label falls within the local vocabulary range managed by the first computing device. For example, the target label exists in the form of an index, with a value range from 0 to the global vocabulary size minus 1. The local vocabulary range managed by the first computing device is determined by its rank-sum segmentation strategy in the distributed system; for example, it may be responsible for indices 0 to C_local-1, or indices C_local to 2×C_local-1, etc. If the index of the target label is within the range managed by the current device, the corresponding predicted score can be directly retrieved from the read predicted score vector based on the offset of the index within the local range. If the index of the target label is not within the range managed by the current device, it is not necessary to obtain the true value from the local predicted score, but it needs to be processed in a specific way. However, this situation is not limited in this embodiment and can be described with reference to the embodiments below.

[0090] In one aspect of this embodiment, firstly, reading the multiple predicted scores corresponding to the sample is the foundation for subsequent calculations. Whether determining the local maximum and local exponent sum, or obtaining the predicted score corresponding to the target label, the predicted score is required as input data. Without the aforementioned reading operation, subsequent calculations cannot proceed. Secondly, there is no strict sequential dependency between determining the local maximum and local exponent sum and obtaining the predicted score corresponding to the target label. They can be executed in parallel or sequentially. For example, during a single iteration of the predicted scores, the local maximum and local exponent sum can be updated simultaneously (or the local maximum can be updated first, followed by the local exponent sum), and it can be determined whether the currently accessed predicted score corresponds to the target label; alternatively, the calculation of the local maximum and local exponent sum can be completed first, and then the corresponding predicted score can be extracted separately based on the target label index. Finally, the feasibility of obtaining the predicted score corresponding to the target label depends on the determination of the target label's scope. Only by clearly determining whether the target label belongs to the scope of responsibility of the current device can the extraction operation be correctly performed or other alternative measures be taken. This determination can usually be completed based on a simple calculation of the device rank, the segmentation strategy, and the target label index; this application does not limit the specific acquisition operation.

[0091] On the one hand of this embodiment, the process of determining the local maximum value and the local exponential sum can be implemented in various ways. For example, the traversal and comparison method can be adopted: initialize a minimum value as the initial value of the local maximum value, and initialize zero as the initial value of the local exponential sum; then access each prediction score one by one. For the currently accessed prediction score, compare it with the current local maximum value. If it is greater than the current local maximum value, update the local maximum value and adjust the local exponential sum based on the difference between the new and old maximum values. Otherwise, directly accumulate the exponential value of the current prediction score into the local exponential sum. After the traversal is completed, the final local maximum value and the local exponential sum can be obtained. The two-stage method can also be adopted: first traverse once to find the local maximum value, and then traverse a second time to calculate the exponential value of each prediction score relative to this maximum value and accumulate it to obtain the local exponential sum. Each method has its own advantages and disadvantages. The former only requires one traversal and has higher calculation efficiency; the latter has clearer logic and is easier to understand and implement.

[0092] On the one hand of this embodiment, the process of obtaining the prediction score corresponding to the target label is relatively straightforward. The first computing device can directly determine the range of the local vocabulary it is responsible for according to the above input data or the preset segmentation strategy. For example, the starting index is start, and the ending index is start + C_local - 1. For the target label index target, if start ≤ target < start + C_local, the target label belongs to the range of the local vocabulary responsible for the current device. At this time, the offset of the target label in the prediction tensor slice is target - start, and accordingly, the corresponding prediction score can be taken out from the read prediction tensor slice as the prediction score corresponding to the target label of the sample.

[0093] The core of the above process is that for each sample, the local maximum value and the local exponential sum are determined based on multiple prediction scores corresponding to the sample, and the prediction score corresponding to the target label is obtained from multiple prediction scores included in the prediction tensor slice on the premise that the target label belongs to the local range. In practical applications, those skilled in the art can make various variations and adjustments to the specific implementation methods of the above operations according to different hardware architectures, programming models, and other requirements. For example: 1. The way of reading the prediction score can be vectorized loading, cache prefetching, DMA (Direct Memory Access) transfer, etc., not limited to accessing each element one by one; 2. The determination of the local maximum value and the local exponential sum can adopt reduction algorithms, parallel reduction, hardware acceleration instructions, etc., not limited to serial traversal; 3. The calculation of judging the belonging range of the target label can be based on bit operations, look-up table methods, or pre-computed offsets, not limited to simple numerical comparison; 4. When obtaining the prediction score corresponding to the target label, if the target label belongs to the local range, pointer offset, index mapping, etc. can also be adopted, not limited to direct array access.

[0094] It is worth noting that any technical solution that determines the local maximum and local exponent sum based on the predicted score corresponding to the sample, or extracts the predicted score corresponding to the target label when the target label belongs to the local range, regardless of its specific implementation details, falls within the protection scope of this dependent claim.

[0095] By explicitly defining the local maximum and local exponent sum as determined based on multiple predicted scores corresponding to the sample, these two values ​​accurately characterize the local predicted score features contained in the predicted tensor slices handled by the first computing device. The local maximum reflects the peak prediction tendency within a local range, while the local exponent sum aggregates the exponential information of all predicted scores within the local range. Together, they constitute a complete description of the statistical characteristics of the local predicted tensor slices. Simultaneously, by directly obtaining the predicted score corresponding to the target label from the multiple predicted scores corresponding to the sample when the target label belongs to the slice range handled by the first computing device, the consistency between the extracted predicted score and the original predicted data is ensured, avoiding information bias. These technical measures collectively ensure the accuracy of each component in the generated local statistical information, providing a reliable data foundation for subsequent loss calculation based on global statistical information, thereby improving the accuracy of loss value calculation.

[0096] In some embodiments, if the target label of a sample does not belong to the segmentation range handled by the first computing device, the prediction score corresponding to the target label of the sample is determined to be a preset value.

[0097] In one aspect of this embodiment, during the generation of local statistical information, for each sample, the predicted score corresponding to its target label may or may not exist on the first computing device. This is because, in tensor parallel training mode, the global vocabulary is divided into multiple non-overlapping segments and stored separately on various computing devices, with each computing device holding only its corresponding prediction tensor segment. Therefore, for any sample, the predicted score corresponding to its target label must exist and only exist on the computing device responsible for the vocabulary range of that target label, while other computing devices, not holding the prediction tensor segment corresponding to that part of the vocabulary, cannot obtain the true value of the predicted score locally. This embodiment addresses the situation where "the target label of the sample does not belong to the segment range responsible for by the first computing device" by limiting the processing method for the predicted score corresponding to the target label of that sample, that is, determining it as a preset value.

[0098] In one aspect of this embodiment, when the first computing device determines, based on the local vocabulary it is responsible for, that the target label index of the current sample is not within the aforementioned local vocabulary range, it indicates that the predicted score corresponding to the target label of the sample is stored on another computing device. At this time, the first computing device cannot obtain the true value of the predicted score from its local predicted tensor fragment. Although the true value cannot be obtained locally, a predicted score item corresponding to the target label still needs to be retained for each sample in the local statistical information data structure. This is to ensure the uniformity of the local statistical information data structure; that is, regardless of whether the target label of the sample belongs to the local range, the local statistical information consists of three components: the local maximum value, the local index, and the predicted score corresponding to the target label. This unified data structure facilitates subsequent aggregated communication operations and global statistical information aggregation, ensuring that the data sent and received by each computing device has the same format and length, eliminating the need to design variable-length data structures or complex index mappings for different situations.

[0099] In one aspect of this embodiment, the specific value of the preset value can be selected according to the needs of subsequent calculations. The core principle is that it should be correctly identified in subsequent global statistical information processing and should not make an incorrect contribution to the final loss calculation result. Since the preset value does not represent the actual predicted score, in subsequent loss calculations, it should be ensured that the preset value does not participate in the calculation like the actual value, or that its participation in the calculation is such that its influence can be correctly offset or ignored. Specifically, the value of the preset value can be flexibly selected according to the specific implementation method of the subsequent loss calculation. Several exemplary value selection methods are provided below: 1. Negative Infinity or Minimal Value: If the subsequent loss calculation uses the cross-entropy loss function, the calculation process requires exponential or logarithmic operations on the predicted score corresponding to the target label. In this case, the preset value can be set to negative infinity (or a negative number with a very large absolute value in floating-point representation). Since the exponential value of negative infinity approaches zero, and negative infinity of the logarithm is undefined, in actual calculations, by combining it with the global maximum value for offset processing, this preset value can be effectively masked in subsequent global exponential sum calculations and loss calculations. Specifically, when calculating the global exponential sum, the preset value from devices other than the target label is negligible in its contribution to the exponential sum due to its extremely small value; when calculating the loss value, only the true predicted score on the target label device is used in the final calculation, while the preset values ​​on other devices do not affect the correctness of the loss result.

[0100] 2. Zero Value: In some implementations, the preset value can be set to zero. In this case, it needs to be adapted to subsequent calculation logic. For example, if the predicted score corresponding to the target label needs to be subtracted or added when calculating the loss value, the zero value, as a neutral element, will not shift the calculation result. However, it should be noted that if subsequent division or exponential operations are involved, the zero value may produce unexpected results. Therefore, it is necessary to ensure that the calculation process can distinguish between the preset value and the true value.

[0101] 3. Flags or Sentinel Values: In addition to the numerical preset values ​​mentioned above, special flags or sentinel values ​​can be used to indicate that "the target label is not local." For example, a special floating-point number (such as infinity) outside the normal range of predicted scores can be used, or a separate flag field can be reserved. However, in actual engineering implementations, to maintain the simplicity of the data structure and the efficiency of computation, it is usually preferable to use preset values ​​such as negative infinity, which can directly participate in numerical calculations and will not have an erroneous impact on the final result.

[0102] It should be noted that regardless of the preset value used, the core purpose is to accurately convey the information that "the predicted score corresponding to the target label of this sample is not on this device," while ensuring the uniformity of the local statistical information data structure. This allows subsequent global statistical information aggregation and loss calculation to correctly identify and handle this situation. It should be emphasized that the above descriptions regarding preset values ​​of negative infinity, zero, or flag bits are merely illustrative and do not constitute a limitation on the scope of protection of this application. The core of the above technical solution is that when the target label does not belong to the segmentation range handled by the first computing device, the predicted score corresponding to the target label is determined as a preset value to ensure the uniformity and integrity of the local statistical information data structure.

[0103] In practical applications, those skilled in the art can select any suitable preset value based on the specific algorithm for subsequent loss calculation, numerical accuracy requirements, and hardware platform characteristics. As long as the preset value can be correctly written into the local statistical information and reasonably processed in the subsequent global statistical information aggregation and loss calculation process, so that the final calculated loss value is based only on the predicted score corresponding to the true target label and is not affected by the error of the preset value, it falls within the protection scope of this application.

[0104] By setting the predicted scores for target labels that are not localized to preset values, the data structure of each sample in the local statistics remains complete and consistent, ensuring no data items are missing regardless of which computing device is actually responsible for the target label. This unified data structure facilitates subsequent ensemble communication operations, allowing each computing device to send and receive data in a fixed format without requiring complex index mappings or variable-length data structures for different samples. This simplifies the communication protocol implementation and improves communication efficiency. Furthermore, by appropriately selecting preset values ​​(such as negative infinity), these preset values ​​are correctly masked in subsequent loss calculations, preventing erroneous impacts on the final loss calculation results and ensuring accuracy.

[0105] In some embodiments, reference may be made to Figure 2 , Figure 2 This is a flowchart illustrating the determination of local maxima and local exponential sums provided in one possible implementation of this application. The method for determining local maxima and local exponential sums can be based on multiple predicted scores corresponding to a sample. Specifically, it may include at least one of steps 210-240: Step 210: Iterate through the multiple predicted scores corresponding to the samples.

[0106] In one aspect of this embodiment, in determining the local maximum and local exponential sum of each sample, it is first necessary to access and process the multiple predicted scores corresponding to that sample. Traversal refers to accessing each predicted score corresponding to the sample one by one or in batches according to a certain order or strategy, so as to perform subsequent comparison, calculation, and accumulation operations. Traversal is one of the most basic and common operation modes in data processing, and its specific implementation method can be flexibly selected according to hardware architecture, data scale, and performance requirements. Traversing the multiple predicted scores corresponding to a sample can cover a variety of different implementation forms, and this embodiment does not make a specific limitation on this, as long as it can complete the access to all predicted scores. For example, the traversal method can include, but is not limited to, the following: 1. Serial Traversal: Serial traversal is the most basic traversal method, in which a single processing unit visits each predicted score corresponding to a sample one by one according to the index order. This method is simple to implement, has clear logic, and is suitable for scenarios with limited processing unit resources or a small number of samples. During serial traversal, it is usually necessary to maintain a pointer or index of the current access position. After each predicted score is accessed, the pointer moves forward until all predicted scores have been accessed.

[0107] 2. Parallel Traversal: With the development of hardware technology, modern computing devices (such as GPUs and multi-core CPUs) typically possess powerful parallel processing capabilities. On such hardware, parallel traversal can be employed, where multiple processing units (such as threads, thread bundles, and cores) simultaneously access different predicted scores corresponding to a sample. Parallel traversal can improve the speed of data access and processing, and is particularly suitable for scenarios with a large number of predicted scores (i.e., a large local vocabulary). For example, specific implementations of parallel traversal may include: evenly distributing the predicted scores among multiple threads, with each thread responsible for accessing a portion of the predicted scores; or using vectorized instructions to load multiple predicted scores for processing at once.

[0108] 3. Block Traversal: Block traversal is a compromise or combination of serial and parallel traversal. It divides the multiple predicted scores corresponding to a sample into several consecutive data blocks. Each data block can be processed in parallel, while blocks are processed sequentially. This approach fully utilizes parallel computing capabilities while avoiding resource contention or cache pressure caused by loading all data (i.e., predicted scores) at once.

[0109] 4. Adaptive Traversal: Adaptive traversal refers to dynamically adjusting the traversal strategy based on hardware resource status, data characteristics, or runtime information. For example, the degree of parallelism can be dynamically determined based on the number of currently available threads, or the traversal can be terminated early based on the numerical distribution of the predicted scores (e.g., after finding a sufficiently large maximum value, there is no need to continue traversing all predicted scores).

[0110] It should be noted that traversal itself is merely a means of data access, not the ultimate goal. In this embodiment, traversal is used to synchronously or asynchronously perform subsequent numerical comparisons, maximum value updates, exponentiation, and accumulation operations while accessing each predicted score. Therefore, the specific implementation of traversal is often tightly coupled with the implementation of subsequent calculations. For example, in serial traversal, maximum value updates and exponentiation / accumulation are usually executed sequentially within the same loop body as the traversal operation; while in parallel traversal, it may be necessary for each thread to complete local calculations first, and then merge the results through reduction operations.

[0111] In a specific embodiment based on a unified computing device architecture (i.e., a programming model that supports languages ​​such as C, C++, and Python to achieve CPU and GPU collaborative computing), step 210 relies on the parallel computing capabilities of the GPU, using a parallel traversal approach to access the predicted scores corresponding to each sample. Specifically, the above embodiment can adopt a scheduling strategy with thread bundles as the basic processing unit. Each thread bundle contains 32 threads, which are assigned to process a row of predicted scores (of length C_local) corresponding to a sample. Within this thread bundle, the 32 threads work collaboratively in a single instruction multiple data stream (SMILE) manner to jointly complete the parallel traversal of all predicted scores for that sample. To achieve the above efficient parallel traversal, the following techniques can be used: 1. Vectorized loading: Threads in the thread bundle use vectorized memory loading instructions to read multiple consecutive predicted scores from global video memory at once, thereby reducing the number of memory accesses and improving video memory bandwidth utilization; 2. Cooperative access: The 32 threads can work collaboratively to cover the entire row of predicted scores. For example, C_local predicted scores can be evenly distributed among the 32 threads, with each thread responsible for accessing several consecutive elements within them. When C_local is not divisible by 32, some threads may handle an additional element, or process the remaining part through a loop; 3. Combination with the online Softmax algorithm: While traversing in parallel, each thread performs local calculations of the online Softmax algorithm on the predicted score it is responsible for, including comparison with the current maximum value, exponentiation, etc. These calculations are completed at the register level, without accessing global video memory, achieving a high degree of overlap between calculation and memory access. Through the above parallel traversal method, the first computing device can complete the access to all predicted scores of the sample in a short time, laying the data access foundation for subsequent calculations of local maximum values ​​and local exponents.

[0112] It is important to emphasize that the parallel traversal method based on thread bundles and vectorized loading described above is merely an exemplary implementation of this embodiment and does not constitute a limitation on the scope of protection of this step. The core of this step lies in the overarching concept of "traversing multiple prediction scores corresponding to samples," rather than specific parallel strategies, hardware platforms, or programming models. In practical applications, those skilled in the art can make corresponding variations and adjustments to the traversal method according to different hardware architectures, different programming frameworks (such as OpenCL (Open Computing Language), etc.), and different performance optimization requirements. For example: 1. On GPUs, multi-threaded parallel traversal can be used, with each thread processing a portion of the prediction scores; 2. On processors supporting SIMD instruction sets, vector instructions can be used to process multiple data elements at once; 3. For sparse prediction scores, skip traversal or index traversal can be used, accessing only non-zero or valid elements; 4. For streaming processing scenarios, pipelined traversal can be used, overlapping data loading and computation. It is worth noting that any technical solution based on the core concept of step 210, namely, accessing multiple predicted scores corresponding to a sample through traversal to provide a data basis for subsequent determination of local maxima and local indices, regardless of its specific traversal strategy, parallel granularity, or hardware implementation, falls within the protection scope of this step.

[0113] Step 220: If the predicted score of the current visit and the current local maximum value satisfy the first preset condition, then based on the predicted score of the current visit and the current local maximum value, multiply the current local exponent sum to obtain the updated local exponent sum; and use the predicted score of the current visit as the updated local maximum value.

[0114] Step 230: If the predicted score of the current visit and the current local maximum value satisfy the second preset condition, then based on the predicted score of the current visit and the current local maximum value, perform cumulative calculation on the current local exponent sum to obtain the updated local exponent sum; and use the current local maximum value as the updated local maximum value.

[0115] In one aspect of this embodiment, during the process of traversing multiple predicted scores corresponding to a sample, different update strategies are employed to dynamically maintain the local maximum value and the local exponential sum based on the relationship between the currently accessed predicted score and the currently maintained local maximum value. The aforementioned relationship includes a first preset condition and a second preset condition, and the update methods for the local exponential sum and the local maximum value are defined for different conditions. This dynamic update mechanism based on conditional branching is the core logic for simultaneously completing the maximum value search and exponential sum accumulation during a single traversal.

[0116] In one aspect of this embodiment, the first and second preset conditions are two mutually exclusive scenarios set based on the numerical comparison between the currently accessed predicted score and the current local maximum value. During the traversal, for each newly accessed predicted score, one and only one of these conditions must be satisfied. This mutual exclusion relationship ensures the determinism and completeness of the computational logic. Specifically, the first preset condition corresponds to the scenario where "the currently accessed predicted score is greater than the current local maximum value." When this condition is met, it means that a new, larger peak value has been encountered during the traversal, and the local exponent sum maintained based on the old maximum value needs to be corrected according to the difference between the new and old maximum values ​​in order to continue accumulating the exponent contribution of subsequent elements. The second preset condition corresponds to the scenario where "the currently accessed predicted score is less than or equal to the current local maximum value." When this condition is met, it means that the current maximum value is still valid, and the newly accessed predicted score can be directly exponentially calculated based on the current maximum value and accumulated into the local exponent sum without needing to correct the existing exponent sum. Exemplarily, the specific implementation of the first and second preset conditions can be numerical comparison operations, such as greater than (>) judgment and less than or equal to (≤) judgment. However, under different numerical representations or calculation precision requirements, other equivalent forms can also be used, such as combinations of greater than or equal to (≥) and less than (<), or an approximate comparison with a tolerance range can be introduced. This embodiment does not strictly limit the specific implementation form of the preset conditions, as long as it can distinguish whether the predicted score of the newly accessed data is greater than the current local maximum value.

[0117] In one aspect of this embodiment, when the currently accessed predicted score and the current local maximum value satisfy a first preset condition (i.e., the current predicted score is greater than the current local maximum value), the following operations can be performed: 1. Perform a product operation on the current local exponential sum: Since the previously accumulated local exponential sum was calculated based on the old local maximum, and the new local maximum is larger, the old local exponential sum needs to be multiplied by a correction factor to convert it into an exponential sum based on the new maximum. For example, the correction factor could be: exp(old local maximum - new local maximum). Through this product operation, the exponential sum originally based on the old local maximum is adjusted to a value aligned with the new local maximum, thus allowing it to be correctly accumulated with subsequent exponential contributions based on the new local maximum.

[0118] 2. Obtain the updated local exponential sum: After completing the above product operation, it is also necessary to include the exponential contribution of the currently visited prediction score itself. Since the current prediction score is equal to the new local maximum, its exponential value relative to the new local maximum is: exp(new local maximum - new local maximum) = exp(0) = 1. Therefore, the result of the product operation can be incremented by 1 to obtain the updated local exponential sum.

[0119] 3. Use the currently visited predicted score as the updated local maximum: Since the current predicted score is greater than the old local maximum, the current predicted score can become the new local maximum, and subsequent traversals will use this value as a basis for further comparison and calculation.

[0120] In one aspect of this embodiment, when the currently accessed predicted score and the current local maximum value satisfy a second preset condition (i.e., the current predicted score is less than or equal to the current local maximum value), the following operations can be performed: 1. Accumulate the current local exponent sum: Since the current local maximum is still valid, newly accessed predicted scores can be directly exponentially calculated based on the current local maximum, and the result is accumulated into the existing local exponent sum. The exponent calculation is: exp(current predicted score - current local maximum), and the result is a positive number less than or equal to 1.

[0121] 2. Obtain the updated local exponential sum: Add the results of the above exponential calculations to the current local exponential sum to obtain the updated local exponential sum. The original local exponential sum remains unchanged; only the new contribution is added to it.

[0122] 3. Use the current local maximum as the updated local maximum: Since the current predicted score has not exceeded the current maximum, the local maximum remains unchanged and does not need to be updated.

[0123] It is worth noting that steps 220 and 230 execute different update paths based on the relationship between the current predicted score and the current local maximum, but the ultimate goal is the same: after traversing all predicted scores, obtain the correct local maximum and the local exponential sum based on that maximum. The alternating execution of the above two steps allows this embodiment to complete the calculation of the local maximum and the local exponential sum in a single traversal, without additional storage and secondary traversal.

[0124] To facilitate understanding of steps 220 and 230 above, a complete implementation based on a unified computing device architecture programming model and online algorithm is given below. Exemplarily, in a thread bundle processing a sample, each thread is responsible for accessing a partial prediction score and maintaining its local maximum and exponential sum. However, at the thread bundle level, it is also necessary to obtain the final local maximum and local exponential sum of the sample through reduction operations. The execution process of steps 220 and 230 is illustrated below from the perspective of a single thread, with specific numerical examples. Specifically, it includes: 1. Initialization: Let the predicted scores accessed by the current thread be 2.0, 5.0, 1.0, and 3.0 respectively (this is just an example; in practice, each thread can be responsible for more elements); initialize the local maximum value (let's call it local_max) local_max=-inf (i.e., negative infinity), and the local exponential sum (let's call it local_sum) local_sum=0.

[0125] 2. Iterate through the first element x1 = 2.0: Currently, local_max = -inf, and x1 > local_max is true (the first preset condition is met), so proceed to step 220: local_sum=local_sum×exp(local_max-x1)+1=0×exp(-inf-2.0)+1=1; local_max=x1=2.0; At this point, local_max = 2.0 and local_sum = 1.

[0126] 3. Iterate through the second element x2 = 5.0: Currently, local_max = 2.0, and x2 > local_max is true (the first preset condition is met), so proceed to step 220: local_sum=local_sum×exp(local_max-x2)+1=1×exp(2.0-5.0)+1=1×exp(-3.0)+1≈1×0.0498+1=1.0498; local_max=x2=5.0; At this point, local_max = 5.0 and local_sum = 1.0498.

[0127] 4. Iterate through the third element x3 = 1.0: Currently, local_max = 5.0, and x3 ≤ local_max is true (the second preset condition is met), so proceed to step 230: local_sum=local_sum+exp(x3-local_max)=1.0498+exp(1.0-5.0)=1.0498+exp(-4.0)≈1.0498+0.0183=1.0681; local_max=x2=5.0; At this point, local_max = 5.0 and local_sum = 1.0681.

[0128] 5. Iterate through the fourth element x4 = 3.0: Currently, local_max = 5.0, and x4 ≤ local_max is true (the second preset condition is met), so proceed to step 230: local_sum=local_sum+exp(x4-local_max)=1.0681+exp(3.0-5.0)=1.0681+exp(-2.0)≈1.0681+0.1353=1.2034; At this point, local_max = 5.0 and local_sum = 1.2034.

[0129] The local maximum (local_max) for this thread is 5.0, and the local exponent sum (local_sum) is approximately 1.2034. This result will be used for subsequent thread-bound reduction to obtain the local maximum and local exponent sum for the entire sample. In the above process, steps 220 and 230 are executed alternately, dynamically updating the local maximum and local exponent sum based on the relationship between the predicted score of each visit and the current local maximum, ultimately completing all calculations in a single traversal.

[0130] It should be emphasized that the specific calculation formulas and numerical examples based on the online algorithm described above are merely an exemplary implementation of this embodiment and do not constitute a limitation on the scope of protection of this application. The core of this embodiment lies in: updating the local exponent sum by using product operations or cumulative operations respectively, based on the different conditions satisfied by the predicted score of the current visit and the current local maximum, and determining whether to update the local maximum value accordingly.

[0131] In practical applications, those skilled in the art can modify and adjust specific calculation methods according to different numerical precision requirements, hardware characteristics, and optimization goals. For example: 1. The exponent base in product operations is not limited to the natural constant e; other values ​​can also be used, as long as they remain consistent in subsequent calculations; 2. For the processing when the first preset condition is met, in addition to the above-mentioned "multiply first and then add 1" method, other equivalent transformations can be used, or the exponent value can be approximated by using methods such as table lookup; 3. For the processing when the second preset condition is met, the accumulation operation can be optimized using fused multiplication-addition instructions to improve computational efficiency; 4. In low-precision calculation scenarios, scaling factors or dynamic range adjustments can be introduced to avoid precision loss.

[0132] It is worth noting that any technical solution based on the core concept of this embodiment, namely, processing the cases where the predicted score is greater than the current local maximum value and the cases where it is not greater than the current local maximum value through conditional branches, and updating the local index and the local maximum value accordingly, regardless of its specific mathematical form, computational precision or optimization method, falls within the protection scope of this application.

[0133] Step 240: After accessing the last predicted score among the multiple predicted scores corresponding to the sample, determine the local maximum and local exponential sum corresponding to the sample based on the updated local maximum and the updated local exponential sum.

[0134] In one aspect of this embodiment, during the process of traversing multiple predicted scores corresponding to a sample, steps 220 and 230 dynamically update the local maximum value and local exponent sum based on the relationship between the predicted score accessed each time and the current local maximum value. However, the above updates are all intermediate results obtained based on the accessed partial data, rather than the final determined statistical information. Step 240 further clarifies that only after all predicted scores corresponding to the sample have been accessed, i.e., at the end of the traversal process, can the updated local maximum value and updated local exponent sum maintained at this time be determined as the final local maximum value and local exponent sum of the sample. Specifically, regardless of the traversal method used (serial, parallel, block, etc.), a traversal completion flag needs to be set or a loop condition needs to be used to ensure that all predicted scores have been processed. When the traversal end condition is met (e.g., the index reaches C_local-1 in serial traversal, or all threads in parallel traversal complete the processing of their respective parts and complete the reduction), the currently held local maximum value and local exponent sum are the final results. It should be noted that in the parallel traversal scenario, there may be multiple processing units that maintain their own local maximum values ​​and local exponent sums respectively. At this point, "accessing all predicted scores" not only means that each processing unit has completed the traversal of its responsible part, but also that the scattered intermediate results need to be merged into a unified final result through reduction operations.

[0135] It is important to emphasize that the core of step 240 is that the updated local maximum and the updated local exponent sum held at this point can only be determined as the final local statistical information after all predicted scores corresponding to the sample have been accessed. In practical applications, those skilled in the art can make various variations and adjustments to the process of "determining the result after accessing all predicted scores" based on different parallel models, hardware platforms, and algorithms. For example: 1. In a CPU multi-threaded environment, atomic operations or mutexes can be used to protect shared variables, and the main thread can read the final result after all threads have completed; 2. On processors that support the SIMD instruction set, the merging of the final results can be completed through vector register reduction instructions; 3. For streaming processing scenarios, a completion flag can be set, and the final result calculation can be triggered after all data blocks have been processed; 4. In heterogeneous computing systems, the host can synchronize the calculation completion status of each device and then obtain the final result uniformly.

[0136] It is worth noting that any technical solution based on the core concept of this embodiment, namely, ensuring that the local maximum and local exponent sum corresponding to the sample are determined only after a complete traversal of all predicted scores, regardless of its specific reduction method, synchronization mechanism or hardware implementation, falls within the protection scope of this step.

[0137] By traversing multiple predicted scores corresponding to a sample, and dynamically updating and maintaining the local index sum and local maximum value using product or accumulation operations based on different preset conditions satisfied by the currently accessed predicted score and the current local maximum value during the traversal, the local maximum value and local index sum corresponding to the sample are determined after all predicted scores have been accessed. This technical solution can simultaneously complete the local maximum value search, local index sum accumulation, and extraction and attribution determination of the predicted score corresponding to the target label in a single traversal, without requiring multiple independent data accesses for different computational purposes. In summary, this embodiment reduces the number of data reads and computational overhead while ensuring the accuracy of local statistical information, shortens the time for acquiring local statistical information, and thus improves the training efficiency of distributed training.

[0138] In some embodiments, based on global statistical information, at least one loss value is generated for each sample, including: Based on global statistical information, determine the global maximum value and the global exponent sum; for each sample, determine the corresponding loss value based on the predicted score corresponding to the sample's target label, the global maximum value, and the global exponent sum.

[0139] In some embodiments, global statistics include multiple local statistics corresponding to a sample. Determining the global maximum value and the global exponent sum based on the global statistics includes: For each sample, based on global statistical information, the multiple local maxima corresponding to the sample are reduced to obtain the global maximum value corresponding to the sample; based on the global maximum value, the multiple local exponential sums corresponding to the sample are corrected; the corrected multiple local exponential sums are reduced to obtain the global exponential sum corresponding to the sample.

[0140] In one aspect of this embodiment, the global statistics are a complete set of local statistics for each sample across all computing devices. Specifically, for any sample, its global statistics include a triplet of local statistics from each computing device, namely, the local maximum value, the local exponent sum, and the predicted score corresponding to the target label stored (or preset) on that device. The above statistics collectively constitute the data basis for calculating the global maximum value and the global exponent sum for that sample. Exemplarily, the global statistics can be organized independently for each sample. For example, for sample i, its global statistics can be represented as: global statistics [i] = {(local_max_{i, 0}, local_sum_{i, 0}, local_logit_{i, 0}), (local_max_{i, 1}, local_sum_{i, 1}, local_logit_{i, 1}), ..., (local_max_{i, W-1}, local_sum_{i, W-1}, local_logit_{i, W-1})}. Where W represents the number of computing devices in the distributed training system, local_max_{i, j} represents the local maximum value of sample i (where i is less than or equal to the total number of samples N) on the j-th computing device (where j is less than or equal to W), local_sum_{i, j} represents the local exponential sum of sample i on the j-th device, and local_logit_{i, j} represents the predicted score corresponding to the target label of sample i on the j-th device (if the target label belongs to the range of this device, it is the true value; otherwise, it is the preset value). This structured organization allows the first computing device to access local statistical information from all devices on a sample-by-sample basis, facilitating subsequent loss calculation.

[0141] In one aspect of this embodiment, for each sample, the first computing device needs to determine the global maximum value and global exponent sum of the sample across all computing devices from its global statistics. For example, taking the aforementioned global statistics [i] as an example, this process can be implemented through the following steps: 1. Reduce multiple local maxima to obtain the global maximum: For sample i, the first computing device reads the local maximum values ​​from all computing devices, local_max_{i, 0} to local_max_{i, W-1}, and finds the maximum value from them through a reduction operation. For example, the reduction operation can be implemented through successive comparisons, parallel reduction, or by calling a reduction primitive provided by the underlying communication library. Regardless of the method used, the global maximum value (denoted as global_max_i) should satisfy: global_max_i = max(local_max_{i, 0}, local_max_{i, 1}, ..., local_max_{i, W-1}). This global maximum value reflects the highest prediction score for sample i across all computing devices.

[0142] 2. Based on the global maximum value, correct the sum of multiple local exponents: Since the local sum of each computing device, `local_sum_{i,j}`, is calculated based on its respective local maximum value, `local_max_{i,j}`, and the global maximum value, `global_max_i`, may differ from `local_max_{i,j}`, it is necessary to correct the sum of the exponents of all computing devices to correctly accumulate their contributions. This requires correcting the local sum of each device to an exponent sum based on the global maximum value (denoted as `local_sum_{i,j}^{corrected}`). For example, this correction can be: `local_sum_{i,j}^{corrected} = local_sum_{i,j} × exp(local_max_{i,j} - global_max_i)`. Essentially, this correction adjusts the original exponent sum based on `local_max_{i,j}` to an exponent sum based on `global_max_i`.

[0143] 3. Reduce the sum of the modified local exponents to obtain the global exponent sum: After completing the local exponent sum corrections for all computing devices, the first computing device can perform summation reduction on multiple corrected local exponent sums to obtain the global exponent sum (denoted as global_sum_i) for sample i. For example, the above summation reduction method can be: global_sum_i = local_sum_{i, 0}^{corrected} + local_sum_{i, 1}^{corrected} + ... + local_sum_{i, W-1}^{corrected}. The global exponent sum is the sum of the indexed prediction scores of sample i across all computing devices, i.e., the complete value of the Softmax normalized denominator.

[0144] In one aspect of this embodiment, after obtaining the global maximum value `global_max_i`, global exponent, and global_sum_i for each sample, the first computing device can further utilize the predicted score corresponding to the target label of the sample to calculate the loss value of the sample. For sample i, the predicted score `logit_i` corresponding to its target label needs to be obtained from the global statistics. It should be noted that although the global statistics contain the predicted scores `local_logit_{i, j}` corresponding to the target label from all devices, only the value on one computing device is the true predicted score (i.e., the computing device where the target label is located), and the values ​​on other computing devices are preset values ​​(such as negative infinity). In a specific implementation, `local_logit_{i, j}` from all computing devices can be directly extracted from the global statistics, but since only one is a valid value, it is usually necessary to combine the target label attribution information to determine which value should be used. For example, a simplified approach is as follows: since preset values ​​(such as negative infinity) do not substantially affect the results in subsequent calculations, the sum (or maximum value) of local_logit_{i, j} on all devices can be directly calculated to automatically obtain the true value (because only the true value is valid, and the contributions of other preset values ​​are negligible). Another approach is: each computing device, when generating local statistics, has already recorded whether it holds the predicted score corresponding to the true target label. In subsequent calculations, this can be directly identified through the device index or flag. After obtaining logit_i, the loss value of the sample can be calculated based on the cross-entropy loss function. For example, the above calculation formula can be: loss_i = -(logit_i - global_max_i - log(global_sum_i)). The derivation of this formula is based on the definition of the Softmax function and logarithmic transformation. By subtracting global_max_i and taking the logarithm, the numerical stability of the calculation process is ensured, avoiding overflow problems that may occur with direct calculation. It is worth noting that for valid samples (such as samples at non-filled positions), the loss_i calculated by the above formula is the final loss value for that sample. For invalid samples, the loss value can be set to zero directly, or they can be excluded in subsequent steps using a masking mechanism.

[0145] In a specific embodiment of a programming model based on a unified computing device architecture, the process of determining the global maximum value, global exponent, and loss value can be accomplished by the underlying kernel. This kernel performs parallel processing on a sample-by-sample basis. For each sample, the kernel first reads the local statistics from all computing devices corresponding to that sample from the global statistics buffer. This data, obtained through aggregate communication operations in step 130, can be stored sequentially in video memory according to the order of the computing devices. Specifically, the kernel execution process includes, but is not limited to, the following steps: 1. Reduction to obtain the global maximum value: The kernel performs parallel reduction on the W local maximum values ​​read through shared memory to obtain the global maximum value of the sample, global_max.

[0146] 2. Correcting and reducing to obtain the global exponential sum: The kernel iterates through the local statistics of all computing devices. For each computing device, it calculates the correction factor exp(local_max - global_max) and multiplies it by the local exponential sum of that computing device to obtain the corrected local exponential sum. Then, these corrected local exponential sums are accumulated to obtain the global exponential sum global_sum.

[0147] 3. Determine the predicted score corresponding to the target label: The kernel iterates through the local statistics of all computing devices to find the predicted score corresponding to the target label. Since this value is only valid on the computing device that actually holds the target label, and is a preset value (negative infinity) on other devices, the maximum value can be directly taken from the local_logit of all devices to obtain the correct predicted score corresponding to the target label. The above method does not require additional flag checks, simplifying the implementation.

[0148] 4. Calculate the loss value: The loss value of the sample can be calculated using the formula: loss = -(logit - global_max - log(global_sum)).

[0149] 5. Handling Invalid Samples: In the above embodiments, the validity of samples can also be determined. For invalid samples (e.g., padding positions in variable-length sequences), their loss values ​​should not participate in the final loss aggregation. In this case, the loss value of the corresponding sample can be set to zero according to the pre-input mask information, or it can be excluded by masking in subsequent loss reduction steps. For example, if a sample is invalid, its corresponding calculation formula is adjusted to: loss = 0.

[0150] The aforementioned kernel completes global maximum value reduction, global exponent calculation, and loss value generation simultaneously in a single call, without requiring multiple kernel startups or data round trips, achieving a high degree of integration in the calculation process.

[0151] It should be emphasized that the above-described specific implementation based on the underlying kernel is merely an exemplary embodiment of this application and does not constitute a limitation on the scope of protection of this application. The core of this embodiment lies in the concept of determining the global maximum value and global exponent sum based on global statistical information, and then determining the sample loss value based on the predicted score corresponding to the target label, the global maximum value, and the global exponent sum. In practical applications, those skilled in the art can make various modifications and adjustments to the above process according to different hardware platforms, programming frameworks, and optimization requirements. For example: 1. Different reduction algorithms can be used to determine the global maximum value and global exponent sum; 2. The exponent operation when correcting the local exponent sum can use a lookup table method, polynomial approximation, or hardware instructions, and is not limited to direct calculation; 3. The extraction of the predicted score corresponding to the target label can be done using various methods such as flag bits, device indexes, or maximum value selection; 4. The formula for calculating the loss value can be adjusted according to different loss function types.

[0152] It is worth noting that any technical solution based on the core concept of this embodiment, namely, extracting the global maximum value and global exponent sum from global statistical information and calculating the loss value by combining it with the predicted score corresponding to the target label, regardless of its specific reduction method, numerical calculation process or optimization strategy, falls within the protection scope of this application.

[0153] By reducing the local maxima of each computing device from the global statistical information to obtain the global maximum, and then correcting the local exponential sums of each device based on the global maximum before reducing to obtain the global exponential sum, the accurate calculation of the global maximum and global exponential sum is ensured. Based on this, the loss value for each sample can be determined by combining the predicted score corresponding to the target label, the global maximum, and the global exponential sum. This technical solution allows the calculation of the loss value to be based entirely on the global statistical information, eliminating the need to backtrack to the original prediction tensor fragments or generate additional intermediate tensors during the calculation process, thus significantly reducing the amount of data to be processed and lowering memory usage. Furthermore, since the data size of the global statistical information is only related to the number of samples and decoupled from the vocabulary size, the loss value calculation process remains efficient even with a large vocabulary, further improving the training efficiency of distributed training.

[0154] In some embodiments, the distributed training method proposed in this application further includes: determining the gradients of multiple predicted scores corresponding to a sample based on global statistical information, wherein the gradients are used to reflect the degree of influence of changes in predicted scores on the loss value.

[0155] In one aspect of this embodiment, during the deep learning training process, after forward propagation calculates the loss value, backpropagation is required to calculate the gradient of the model parameters, thereby updating the model weights. For the cross-entropy loss function, backpropagation requires calculating the gradient corresponding to each predicted score. This embodiment further illustrates that the gradients of multiple predicted scores corresponding to a sample can be determined based on global statistical information. The aforementioned gradient refers to the partial derivative of the loss function with respect to each predicted score, used to reflect the degree of influence of changes in the predicted score on the loss value. In a distributed training scenario, since each computing device only holds a local slice of the prediction tensor, each computing device only needs to calculate the gradient corresponding to the portion of predicted scores it is responsible for, without needing to calculate the global complete gradient tensor. This embodiment is based on this idea, utilizing global statistical information to directly complete the gradient calculation locally.

[0156] In one aspect of this embodiment, the gradients of multiple predicted scores corresponding to a sample are determined based on global statistical information. The core principle is that the gradient expression of the cross-entropy loss function with respect to the predicted score can be completely derived from the global maximum value, global exponent sum, and the target label attribution of the sample, without needing to backtrack to the original prediction tensor segmentation or re-perform complex Softmax calculations. Specifically, for any sample, the gradient corresponding to its predicted score (let's say x) can be expressed as: grad(x) = exp(x - global_max) / global_sum. Here, global_max is the global maximum value of the sample, and global_sum is the global exponent sum of the sample. The derivation of the above formula is based on the differentiation rule of the Softmax function and employs the same numerical stabilization process as the loss calculation (subtracting the global maximum value), ensuring the numerical stability of the calculation process.

[0157] In some embodiments, global statistics include a global maximum and a global exponential sum. Determining the gradient of multiple predicted scores corresponding to a sample based on the global statistics may include: For each of the multiple predicted scores corresponding to a sample, the gradient corresponding to the predicted score is determined based on the global maximum value and the global exponent sum; if the target label of the sample belongs to the segment range responsible for the first computing device, the gradient of the predicted score corresponding to the target label of the sample is corrected.

[0158] In one aspect of this embodiment, for each predicted score in the predicted tensor slice held by the first computing device, its corresponding gradient can be determined according to the following steps: 1. Obtain global statistical information: First, the first computing device needs to obtain the global maximum value global_max, global exponent, and global_sum for each sample. The above data can be directly read from the results calculated and stored in the previous steps without recalculation; 2. Calculate the gradient base value (denoted as grad_base): For each predicted score x of the current sample, its gradient base value can be calculated according to the formula: grad_base=exp(x-global_max) / global_sum. The gradient base value can reflect the degree of contribution of the predicted score to the loss function without considering the special processing of the target label; 3. Store the gradient: The calculated gradient base value can be written to the storage location corresponding to the predicted score, for example, directly overwriting the value at the corresponding position in the original predicted tensor slice, or writing it to a separate gradient buffer. The above-mentioned in-situ storage or reuse storage method can effectively save GPU memory space.

[0159] In one aspect of this embodiment, the calculation of the aforementioned base gradient value applies to all predicted scores corresponding to the sample. However, for the predicted score corresponding to the target label, its gradient needs further correction to conform to the gradient definition of the cross-entropy loss function. For example, according to the gradient formula of the cross-entropy loss function, for the predicted score corresponding to the target label, its gradient (denoted as grad_target) should be: grad_target = exp(x_target - global_max) / global_sum - 1 in certain aggregation modes, where x_target is the numerical value of the predicted score corresponding to the target label; while for the predicted score not corresponding to the target label, its gradient (denoted as grad_nontarget) is: grad_nontarget = exp(x - global_max) / global_sum. That is, the gradient at the target label position needs further processing on top of the base value, while the gradient of the predicted score at the non-target label position is the base value itself. In another aspect of this embodiment, the correction operation is performed only when the target label of the sample belongs to the slice range handled by the first computing device. This is because only when the target label falls on this device does the first computing device hold the correct predicted score corresponding to the target label, and only then does it need to correct the gradient at that position. If the target label is not within the range of this device, all predicted scores on this device correspond to non-target labels, and no correction operation is required.

[0160] In one aspect of this embodiment, the specific value to be subtracted in the correction operation depends on the aggregation mode of the loss values. In distributed training, loss values ​​typically support different aggregation methods, such as summation or averaging: if summation is used, the loss values ​​of each sample are directly added together to obtain the final loss, and the gradient at the target label position needs to be subtracted by 1 from the base value; if averaging is used, the loss values ​​of each sample are added together and then divided by the total number of samples, and the gradient at the target label position needs to be subtracted by 1 / N from the base value, where N is the total number of samples (or the number of valid samples); if other aggregation methods are used (such as weighted summation), the correction amount also needs to be adjusted accordingly. Therefore, the specific value subtracted in the correction operation is not fixed at 1, but is closely related to the aggregation mode of the loss function.

[0161] In a specific embodiment based on a unified computing device architecture programming model, the gradient calculation and correction process described above is uniformly completed by the underlying kernel in the above embodiment. This kernel can calculate the gradient corresponding to each predicted score in parallel while calculating the loss value during forward propagation. Specifically, the processing flow of the kernel for each sample may include the following steps: 1. Obtain global statistics: The kernel reads the global maximum value (denoted as global_max) and the global exponential sum (denoted as global_sum) of the sample from registers or shared memory. These values ​​were obtained when the kernel calculated the loss value. At the same time, the kernel obtains the aggregation mode used for the current loss calculation, such as summation mode or averaging mode.

[0162] 2. Iterate through the local predicted scores and calculate the gradient: The kernel iterates through all predicted scores (C_local) corresponding to the sample held by the first computing device. For each predicted score x, the following calculation can be performed: grad = expf(x - global_max) / global_sum, to obtain the gradient grad corresponding to the initial predicted score.

[0163] 3. Determine if correction is needed: The kernel can determine the target label (denoted as target_id) of the sample and the vocabulary segmentation range of the first computing device. If target_id falls within the range of the local device, its offset in the local prediction score is further determined as local_offset = target_id - start_idx, where start_idx is the starting index of the prediction tensor segment handled by the first computing device. This offset is used to locate the specific position of the prediction score corresponding to the target label in the local index of the current device.

[0164] 4. Perform corrections based on the aggregation pattern: If the currently processed predicted score is the predicted score corresponding to the target label, the kernel can correct the gradient of the predicted score corresponding to the target label according to a preset aggregation mode. For example, if the summation mode is used, 1 is subtracted from the initial gradient value of the predicted score to obtain the final gradient value; if the averaging mode is used, 1 is subtracted from the initial gradient value of the predicted score and divided by the total number of samples (or the number of valid samples) to obtain the final gradient value. For other predicted scores in the sample, no correction operation is required, and their initial gradient value is the final gradient value. In addition, if the current sample is an invalid sample (e.g., a padding position in a variable-length sequence), the gradients of all predicted scores corresponding to that sample are directly set to 0.

[0165] Through the above process, the kernel can complete the calculation of the gradient corresponding to the predicted score in a single traversal, without the need for additional kernel startup or data transfer.

[0166] It should be emphasized that the above-described implementation based on the underlying kernel is merely an exemplary embodiment of this application and does not constitute a limitation on the scope of protection of this application. The core of this embodiment lies in the concept of determining the gradient of the predicted score corresponding to a sample based on global statistical information, and correcting the gradient when the target label belongs to the local range. In practical applications, those skilled in the art can make various modifications and adjustments to the above process according to different hardware platforms, programming frameworks, and optimization requirements. For example: 1. The exponential operation in the gradient calculation formula can use different mathematical library functions or approximation algorithms; 2. The gradient storage location can be an independent gradient buffer or a reused storage space; 3. The correction amount in the correction operation can be flexibly determined according to different aggregation modes (such as weighted summation, masked averaging, etc.); 4. The timing of the correction operation can be completed synchronously during the traversal process or processed separately after the traversal is completed.

[0167] It is worth noting that any technical solution based on the core concept of this embodiment, namely, calculating the predicted score gradient by utilizing global statistical information and correcting the predicted score according to the target label, regardless of its specific calculation order, storage method or numerical precision, falls within the protection scope of this application.

[0168] By directly calculating the gradient for each predicted score using global statistical information and correcting the gradient of the predicted score when the target label belongs to the local range, the following technical effects are achieved: Gradient calculation is entirely based on global statistical information, eliminating the need for backtracking prediction tensor partitioning and recalculating Softmax normalization, significantly reducing the data processing workload in the gradient calculation stage. Simultaneously, gradient correction based on the target label ensures the accuracy of gradient calculation, enabling backpropagation to update model parameters based on the correct gradient values.

[0169] In some embodiments, the gradients of multiple predicted scores corresponding to a sample can be stored in the storage space corresponding to the prediction tensor slice.

[0170] In one aspect of this embodiment, after calculating the gradients of multiple predicted scores corresponding to each sample, the gradient values ​​need to be stored for subsequent gradient accumulation and parameter updates during backpropagation. This embodiment proposes that the gradient storage location may include storing the gradients of the multiple predicted scores corresponding to the sample in the storage space corresponding to the prediction tensor slice. The storage space corresponding to the prediction tensor slice refers to the storage area in the local memory of the first computing device (such as the GPU's video memory) originally used to store the prediction tensor slice. Since the original predicted scores are no longer needed in subsequent calculations after the forward propagation calculation is completed (unless they need to be retained for a specific purpose), the aforementioned storage space can be used to store the calculated gradient values, thereby achieving storage space reuse.

[0171] In one aspect of this embodiment, there is a natural correspondence between the calculated gradients and the predicted scores: each gradient value corresponds to a predicted score, and the two have identical indices in both the sample dimension and the local vocabulary dimension. Specifically, for the prediction tensor slice held by the first computing device (assuming a shape of N×C_local), the predicted score of the i-th sample at the j-th local vocabulary position (denoted as x_{i, j}) is used to calculate the corresponding gradient value (denoted as grad_{i, j}). Therefore, the gradients and predicted scores have the same shape in their data structures, which makes it possible to reuse the storage space of the prediction tensor slice.

[0172] In one aspect of this embodiment, depending on the actual application scenario and subsequent computational needs, storing the gradient in the storage space corresponding to the prediction tensor slice can take various specific forms, and this application does not strictly limit this. Exemplarily, it can include, but is not limited to, the following methods: 1. In-situ Update: This is the most direct and commonly used storage method. After calculating the gradient corresponding to each predicted score, the gradient value can be directly written into the storage unit originally occupied by that predicted score, overwriting the original predicted score. In this method, the gradient fully reuses the storage space of the predicted tensor slice, without the need to allocate any additional GPU memory. During backpropagation, the gradient value can be directly read from this storage space for subsequent operations. The advantage of in-situ update is that it has the highest GPU memory utilization and is suitable for most standard training scenarios.

[0173] 2. Independent Storage: In certain special scenarios, it may be necessary to retain both the original prediction scores and the calculated gradients simultaneously. For example, when the forward propagation results need to be used multiple times for subsequent analysis or calculation, or in some complex training strategies (such as gradient accumulation or recalculation), the original prediction scores may still be useful. In this case, an independent region can be partitioned within the storage space corresponding to the prediction tensor slice, or the prediction scores and gradients can be stored simultaneously in parallel within adjacent contiguous storage spaces. For example, two storage regions of the same size can be allocated to store the prediction scores and gradients respectively; alternatively, the prediction scores and gradients can be stored alternately within the same storage region (e.g., each prediction score is followed immediately by its corresponding gradient). Regardless of the specific layout, as long as the gradients are stored within the storage space corresponding to the prediction tensor slice, they fall within the protection scope of this application.

[0174] 3. Incremental Storage: In some implementations, gradient computation may not be completed all at once, but rather in batches or stages. In this case, incremental storage can be used, where each computed gradient is written to the corresponding location in the memory space of the prediction tensor segment until all gradients are filled. This approach is suitable for scenarios with limited computational resources or where computational tasks need to overlap.

[0175] 4. Compressed Storage: For scenarios where low-precision training is feasible, gradients can be compressed before storage, for example, by converting them from 32-bit floating-point numbers to 16-bit floating-point numbers or even lower precision formats. The compressed gradients can still be stored in the original prediction tensor slice's storage space (potentially occupying less space), or in the corresponding compressed storage area. As long as there is a clear index correspondence between the compressed gradients and the prediction tensor slices, and they are stored in the associated storage space, this can be considered a variant implementation of this application.

[0176] In a specific embodiment based on a unified computing device architecture programming model, gradient storage is implemented using the aforementioned "in-situ update" method to maximize GPU memory utilization efficiency. Exemplarily, during the execution of the underlying kernel in the above embodiment, gradient calculation and storage for each sample can be completed synchronously. Specifically, when the kernel iterates through the prediction scores of each sample held by the first computing device, for each prediction score, it can sequentially perform the following operations: 1. Calculate the gradient value based on the current predicted score, the global maximum value of the current sample, and the global exponent sum.

[0177] 2. If the current predicted score is the predicted score corresponding to the target label and the target label is within the range of this device, then the gradient value is corrected.

[0178] 3. Write the calculated gradient value of the current predicted score directly into the original storage unit where the current predicted score was located, and overwrite the original predicted score value.

[0179] Since the predicted scores are no longer needed after forward propagation, the aforementioned in-situ overlay mechanism does not cause information loss. Simultaneously, the gradient values ​​are precisely placed in the same location as the corresponding predicted scores, maintaining the integrity of the data structure and the consistency of the index. After the kernel completes processing all samples, the storage space originally used to store the predicted tensor pieces now contains the calculated gradient values. During subsequent backpropagation, the computing device can directly read the gradients from this storage space for parameter gradient accumulation and updates, without any additional data handling or format conversion. Furthermore, in this embodiment, if the training configuration requires retaining the original predicted scores (e.g., for debugging or specific analysis), a flag can be set to switch the storage mode. In this case, the kernel will copy the original predicted scores to a temporary buffer before calculating the gradients, and then write the gradients back to the original storage space; or it can directly use a separate gradient buffer while retaining the original predicted scores. Regardless of the mode, the gradients are stored in the storage area corresponding to the predicted tensor pieces.

[0180] It should be emphasized that the above-described kernel-based in-situ update method is merely an exemplary implementation of this application and does not constitute a limitation on its scope of protection. The core of this embodiment lies in storing the calculated gradients in the storage space corresponding to the prediction tensor slices, rather than limiting the specific form of gradient storage. In practical applications, those skilled in the art can make various modifications and adjustments to the specific implementation of gradient storage according to different hardware architectures, training frameworks, and performance requirements. For example: 1. In systems supporting unified memory, gradients can be stored in the same memory region where the prediction tensor slices are located, with the system automatically managing data migration; 2. In training strategies using gradient accumulation, a gradient accumulator can be maintained in the storage space corresponding to the prediction tensor slices, and each calculated gradient can be accumulated onto this accumulator; 3. For scenarios using mixed-precision training, high-precision prediction scores and low-precision gradients can be maintained simultaneously, stored in different regions of the same storage space; 4. In scenarios where gradients need to be sent to other computing devices for aggregation, gradient values ​​can be directly read from this storage space for communication, without the need for additional memory copying.

[0181] It is worth noting that any technical solution based on the core concept of this embodiment, namely storing the calculated gradient to the storage space corresponding to the prediction tensor slice in order to achieve memory reuse or maintain data consistency, regardless of its specific storage method, data format or management strategy, falls within the protection scope of this application.

[0182] By directly storing the calculated gradients in the storage space corresponding to the prediction tensor slices, the following technical effects are achieved: On the one hand, since the gradients and prediction scores are completely corresponding in data structure and the storage locations are reused, there is no need to allocate additional storage space for the gradients, thereby effectively reducing the peak storage space occupation during training; on the other hand, the gradients are stored in the same location as the prediction scores, maintaining index consistency and access locality. During backpropagation, the gradient values ​​can be read directly from the original storage space without additional data movement or address mapping, thus improving data access efficiency.

[0183] To better understand the above technical solutions, the following will combine... Figure 3 A specific embodiment is provided to illustrate the complete implementation process of the distributed training method proposed in this application within the PyTorch framework. For example, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the distributed training process provided in one possible implementation of this application. This embodiment takes tensor parallel training of a large language model as its application scenario, encapsulates the entire loss calculation process into a custom automatic differentiation function, and achieves memory-friendly and computationally efficient loss and gradient calculations through the collaborative design of the underlying kernel and upper-layer interfaces—that is, the distributed training method proposed in this application.

[0184] I. Overall Architecture and Interface Design This embodiment encapsulates the entire optimized computation and communication process into a custom automatic differentiation function, named the Distributed Fusion Cross-Entropy Loss Function. This function inherits from `torch.autograd.Function`, a tool in PyTorch used to define custom forward and backward propagation operations. By inheriting from `torch.autograd.Function`, custom operations can be defined, and their forward and backward propagation logic can be explicitly specified. Specifically, as an interface directly called by the user, such as... Figure 3 As shown, the input data may include: 1. Prediction Tensor Sharding: In tensor parallel mode, prediction tensor sharding is a distributed tensor with shape (N, C_local), where N is the product of batch size and sequence length (i.e., the total number of samples), and C_local is the size of the local vocabulary held by the first computing device. This tensor is stored in the GPU memory of the first computing device and includes the prediction scores corresponding to the local vocabulary handled by the first computing device.

[0185] 2. Global Label Tensor: The global label tensor is a tensor of shape (N,) stored in the video memory of the first computing device. Each element in the global label tensor is an integer index, ranging from 0 to C_global-1, indicating the position of the target label of the corresponding sample in the global vocabulary.

[0186] 3. Aggregation mode parameter: Used to characterize the aggregation method of loss values ​​and / or gradients, such as summation mode or averaging mode.

[0187] 4. Communication group set: Used to identify the set of computing devices participating in the current group communication, that is, the communication group formed by the various computing devices participating in distributed training.

[0188] With the above interface definition, users only need to pass in the predicted tensor fragments and global label tensors output by the model, and configure the corresponding aggregation mode parameters and communication group set to start the subsequent calculation process, without having to worry about the underlying distributed partitioning details.

[0189] II. Forward Propagation During forward propagation, the system schedules the two underlying core computing kernels and handles necessary communication operations within the tensor parallel group. For example... Figure 3 As shown, the forward propagation process includes the following three stages: Phase 1: Local Statistical Kernel The local statistics kernel uses thread bundles as the basic processing unit, with each thread bundle responsible for processing the predicted score row corresponding to one sample. The kernel execution flow can be referred to the previous description of the embodiments for steps 210 to 240, which is briefly summarized here: For each sample, the kernel first determines whether the target label corresponding to the sample is valid. If the target label is an invalid padding position, the components in the local statistics information of the sample are directly set to preset invalid values, and the processing ends. If the target label is valid, the kernel uses the online Softmax algorithm to dynamically maintain the local maximum value and the local exponent sum during a single traversal of the predicted scores. During the traversal, for each predicted score, based on its relationship with the current local maximum value, the local exponent sum is updated using either product or accumulation operations, and the local maximum value is updated or maintained accordingly. Simultaneously, the kernel determines whether the target label is located within the local vocabulary partition based on its index. If so, the corresponding predicted score is retrieved as the predicted score corresponding to the target label; otherwise, the predicted score corresponding to the target label is set to negative infinity. After the traversal is complete, the kernel obtains the final local maximum and local exponent sum of the sample through thread-bound reduction. Together with the predicted score corresponding to the target label, they form a triple (local maximum, local exponent sum, predicted score corresponding to the target label) and are written to a temporary buffer.

[0190] Phase 2: Cross-device aggregated communication Since each computing device only has local vocabulary partitioning information, it is necessary to obtain the global maximum value, exponent, and prediction score corresponding to the target label. In this stage, an all-gather operation is performed by calling the distributed communication primitive to aggregate the local triples from each computing device into a global view. For a specific implementation, please refer to the previous description of the embodiment for step 130. After performing the all-gather operation, each computing device obtains global statistical information, which includes local statistical information from all computing devices. Multiple local statistical information can be organized by sample dimension, providing a complete data foundation for subsequent loss and gradient calculations.

[0191] Phase 3: Loss Calculation and Gradient Calculation Kernel The loss calculation and gradient presetting kernel processes data in parallel on a sample-by-sample basis. It receives global statistics generated in the second stage and performs global maximum reduction, global exponent sum calculation, loss calculation, and gradient presetting within the kernel. The kernel execution flow can be found in the previous description of the implementation examples related to step 140 and gradient calculation; a brief overview is provided here: 1. For each sample, the kernel first reads the local maximum values ​​from all devices in the global statistics and reduces them to obtain the global maximum value. Then, it corrects the local exponential sums of each device based on the global maximum value and reduces the corrected local exponential sums to obtain the global exponential sum.

[0192] 2. After obtaining the global maximum value and the global exponent sum, the kernel determines the predicted score corresponding to the true target label of the sample based on the predicted scores of the target labels provided by each device in the global statistics (this can be achieved by reading the maximum value of this score from all computing devices, since this value is only valid on the computing device that actually holds the target label; on other devices, it is negative infinity). Then, the loss value of the sample can be calculated using the formula loss = -(logit - global_max - log(global_sum)). For invalid samples, the loss value can be directly set to zero.

[0193] 3. Simultaneously, the kernel iterates through all predicted scores corresponding to the sample held by the first computing device. For each predicted score, its base gradient value can be calculated using the formula grad = exp(x - global_max) / global_sum. If the target label of the current sample belongs to the range of this device, and the currently iterated predicted score is the predicted score corresponding to the target label, then the gradient is corrected according to a preset aggregation mode. For example, in the summation mode, 1 is subtracted; in the averaging mode, 1 is subtracted and divided by the total number of samples. The corrected gradient value can be directly written to the corresponding storage space in the prediction tensor slice held by this device, achieving in-situ storage of the gradient.

[0194] like Figure 3 As shown, prior to the second stage (cross-device aggregate communication) described above, the system can select an appropriate communication mode based on the current parallel configuration of distributed training. Specifically, the communication mode can include the following two scenarios: 1. No parallel mode When the tensor parallel size is equal to 1, the system operates in single-device mode and / or without tensor parallelism. In this mode, the entire distributed training system contains only one computing device (e.g., the first computing device) participating in the current training task, or although multiple computing devices exist, they are not partitioned along the tensor dimension. In this mode, the local statistical information of each sample already exists completely on a single computing device, eliminating the need for cross-device data exchange. Therefore, the communication control module skips the all-gather operation in the second stage, and the local statistical information generated in the first stage is directly passed as global statistical information to the loss and gradient calculation kernels in the third stage. This mode is suitable for non-tensor parallel training scenarios, such as pure data parallelism or single-card training, ensuring the versatility of this method under different parallel strategies.

[0195] 2. Tensor Parallel Mode When the tensor parallel size is greater than 1, the system operates in multi-device tensor parallel mode. In this mode, the global vocabulary is divided into multiple non-overlapping fragments, stored on different computing devices, with each device holding only a local prediction tensor fragment. Correspondingly, the local statistical information generated in the first stage only reflects the statistical characteristics of the prediction scores for each device and cannot be used alone to calculate the global loss value. Therefore, the system triggers a second-stage all-gather operation to exchange local statistical information among the computing devices, enabling each device to obtain a complete statistical view of all samples across all devices. This mode achieves the aggregation of necessary information through a single aggregation communication, providing a complete global data foundation for subsequent loss calculation.

[0196] It should be noted that the choice of the above communication mode is completely transparent to upper-layer users. Users only need to configure the distributed training environment in a conventional manner (such as specifying the tensor parallel size). The distributed training method provided in this application can automatically adapt the corresponding communication behavior at the underlying level according to the actual configuration, without requiring manual intervention or modification of the code logic by the user.

[0197] like Figure 3 As shown, in the third stage of the forward propagation process, after the loss calculation and gradient calculation kernels have been executed, the forward propagation stage outputs two types of data—the final loss value and the gradient-ready prediction tensor slices. These two types of data are used for subsequent backpropagation. Specifically: 1. Final Loss Value: The final loss value refers to the scalar loss value after aggregation, which is the main result returned to the upper-level caller by the forward propagation method. The loss calculation and gradient calculation kernel in the third stage above calculates the loss value for each sample, resulting in a corresponding loss value. These loss values ​​can form a one-dimensional tensor of shape (N, ), where N is the total number of samples. Subsequently, an aggregation operation is performed according to a preset aggregation mode: if the aggregation mode is set to summation, the loss values ​​of all samples are summed to obtain the final loss value; if the aggregation mode is set to averaging, the loss values ​​of all samples are averaged to obtain the final loss value. After the aggregation operation, the final loss value exists in duplicate on each computing device, meaning each device holds the same scalar loss value.

[0198] 2. Gradient-Ready Prediction Tensor Fragments: Gradient-ready prediction tensor fragments refer to the set of gradient values ​​stored in the original storage space of the prediction tensor fragment after gradient calculation and correction in the third stage. From a data structure perspective, it is still a tensor of shape (N, C_local), with the exact same size and layout as the prediction tensor fragment initially held by the first computing device. However, from a numerical perspective, it no longer stores the prediction scores from the forward propagation process, but rather the gradient values ​​corresponding to each prediction score. Since the gradients are directly written into the storage space originally used to store the prediction scores, storage space reuse is achieved. This tensor formally inherits the storage location and structure of the prediction tensor fragment, and therefore can be called gradient-ready prediction tensor fragments.

[0199] III. Backpropagation like Figure 3 As shown, after the forward propagation is completed and the final loss value and gradient-ready prediction tensor slices are output, the system enters the backpropagation phase to calculate the gradients of the model parameters and update the weights. The backpropagation process in this embodiment fully utilizes the gradient information pre-set in the forward propagation (i.e., the aforementioned gradient-ready prediction tensor slices), simplifying the calculation process. Specifically: In the automatic differentiation framework, the backpropagation process starts from the output node and propagates gradient information backward along the computation graph. For any operator node, the input received by its backpropagation method is called the "upstream gradient," which represents the derivative of the loss function with respect to the operator's output. The upstream gradient is the core information transmitted in the backpropagation chain. Each operator, according to its own forward propagation calculation logic, converts the upstream gradient into the gradient of its own input and continues to propagate it to the operators in the previous layer.

[0200] For the loss function operator implemented in this embodiment, its forward propagation output consists of two parts: the final loss value and gradient-ready prediction tensor pieces. However, in standard automatic differentiation systems, typically only the final loss value participates in the construction of the backpropagation graph as the output of forward propagation. While the gradient-ready prediction tensor pieces are calculated during forward propagation, they do not participate in backpropagation as output nodes; instead, they are passed to the backpropagation stage through context saving. Therefore, the upstream gradient received by this operator is the gradient from the final loss value, i.e., scalar 1 (representing that the derivative of the loss function with respect to itself is 1).

[0201] In this embodiment, the execution flow of backpropagation can be summarized as follows: Step 1: Read context information: First, read the information saved during the forward propagation phase, which mainly includes: the gradient-ready prediction tensor slices (i.e., the gradient values ​​stored in the original prediction tensor slice storage space), and auxiliary information such as aggregation mode and number of samples if necessary.

[0202] Step 2: Receive upstream gradient: Receive the gradient input from upstream, denoted as grad_output. Since this operator is usually used as the output node of the loss function in the computation graph, the value of grad_output is a scalar of 1, representing the derivative of the loss function with respect to itself.

[0203] Step 3: Calculate the input gradient: The input to this operator is a prediction tensor slice, therefore, it is necessary to calculate the derivative of the loss function with respect to each prediction score, i.e., the input gradient (denoted as grad_input). Since the gradient value corresponding to each prediction score has been pre-calculated during the forward propagation stage (hereinafter referred to as the pre-set gradient) and stored in the gradient-ready prediction tensor slice, grad_input can be directly obtained by multiplying the upstream gradient with the pre-set gradient: grad_input = grad_output × pre-set gradient. Since grad_output is usually 1, the above multiplication operation actually directly returns the pre-set gradient value. This design allows the backpropagation method to perform no complex numerical calculations, only simple multiplication operations.

[0204] Step 4: Return the input gradient: Return the calculated input gradient grad_input to the operator of the previous layer for further backpropagation. The shape of grad_input is exactly the same as the prediction tensor slice, i.e., (N, C_local), which contains the final gradient value corresponding to each prediction score.

[0205] On the other hand, although the final loss value is the main result returned to the upper-level caller during forward propagation, its role in backpropagation is mainly reflected in the following two aspects: 1. Starting point of backpropagation: The final loss value is the starting point of the entire backpropagation chain. The system starts with the final loss value, calls its backpropagation, and passes in the upstream gradient grad_output=1. It is this call that triggers the execution of the backpropagation method of this operator.

[0206] 2. The baseline for gradient calculation: The upstream gradient grad_output=1 means "the derivative of the loss function with respect to itself is 1", which provides a baseline for the entire backpropagation process. The gradients of all subsequent parameters are calculated layer by layer based on this baseline using the chain rule.

[0207] It is important to emphasize that although the final loss value, as a scalar, triggers backpropagation, it does not itself contain gradient information. The gradient information for the predicted score has already been pre-calculated during the forward propagation phase and stored in the gradient-ready prediction tensor slices. The backpropagation phase only needs to multiply these pre-set gradients with the upstream gradients. This design shifts the focus of gradient computation from a dedicated backpropagation phase after forward propagation to the forward propagation process itself, achieving a reallocation and optimization of computational resources.

[0208] Through the above design, this embodiment achieves the following technical effects in the backpropagation stage: 1. For gradient calculation, only one multiplication operation is required, and in some implementations, it can even be simplified to directly returning the pre-set gradient, consuming almost no computational resources; 2. Since complex numerical calculations are not involved, there is no need to start a dedicated kernel, avoiding the overhead of kernel startup and context switching; 3. The predicted tensor slices with ready gradients are directly read and used in backpropagation without additional data handling or format conversion, maintaining the locality and efficiency of data access. In summary, this embodiment simplifies the backpropagation process by pre-setting gradients in forward propagation and performing only simple multiplication operations in backpropagation, further improving the training efficiency of distributed training.

[0209] The technical solution proposed in this embodiment encapsulates the entire distributed cross-entropy loss calculation process into a custom PyTorch automatic differentiation function. At the underlying level, it utilizes a local statistical kernel and loss and gradient calculation kernels, combined with a single cross-device aggregate communication operation, achieving deep integration and optimization of computation and communication. This solution fuses multiple discrete operators into two to three customized kernels, significantly reducing the number of kernel startups and global memory access frequency, thus improving computational efficiency. Simultaneously, the amount of communication data is reduced from being related to the global vocabulary size to being only related to the number of samples, significantly reducing both the number of communication operations and the amount of data, effectively lowering cross-device communication overhead. Furthermore, based on PyTorch's automatic differentiation function encapsulation mechanism, this embodiment provides a fully compatible calling interface with the native cross-entropy loss function in the form of a distributed fused cross-entropy loss function. It supports distributed tensor input and distributed automatic differentiation, enabling integration into the existing PyTorch training ecosystem. Users do not need to be aware of the underlying kernel implementation and distributed communication details to replace the original loss function, enjoying memory optimization and efficiency improvements without modifying the model architecture or the main logic of the training loop, thus lowering the barrier to entry.

[0210] It should be noted that, Figure 3 The specific embodiments shown and the implementation methods based on the PyTorch framework and the unified computing device architecture programming model described above are merely illustrative examples of the technical solutions of this application, intended to help those skilled in the art better understand the core concepts of this application, and do not constitute a limitation on the scope of protection of this application. Any technical solution based on the core concept of this application, namely, realizing loss calculation by generating and exchanging local statistical information, regardless of its specific implementation framework, hardware platform, parallel strategy, or optimization method, falls within the scope of protection of this application. Those skilled in the art can make corresponding modifications and adjustments to the various components in the above embodiments according to different training frameworks, hardware platforms, and performance optimization requirements, as long as they do not depart from the core concepts of this application, they should all be considered to fall within the scope of protection of this application.

[0211] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0212] For example, please refer to Figure 4 , Figure 4 This is a structural block diagram of a distributed training device provided in one possible implementation of this application. The aforementioned distributed training device 400 is applied to a first computing device in a distributed training system. The distributed training system includes the first computing device and at least one other computing device. The device may include: a receiving module 410, a local statistics module 420, a global statistics module 430, and a loss calculation module 440.

[0213] The receiving module 410 is used to receive input data, which includes prediction tensor slices and global label tensors. The global label tensor includes target labels corresponding to multiple samples.

[0214] The local statistics module 420 is used to generate local statistical information for at least one sample based on the prediction tensor slices and the global label tensor. The local statistical information for each sample includes: local maxima, local exponents, and the predicted score corresponding to the sample's target label.

[0215] The global statistics module 430 is used to send first local statistical information to other computing devices and receive second local statistical information from other computing devices to obtain global statistical information. The first local statistical information includes local statistical information corresponding to at least one sample generated by the first computing device, and the second local statistical information includes local statistical information corresponding to at least one sample generated by other computing devices.

[0216] The loss calculation module 440 is used to generate loss values ​​for at least one sample based on global statistical information.

[0217] In some embodiments, the local statistics module 420 is further configured to: for each sample, read multiple prediction scores corresponding to the sample contained in the prediction tensor slice, and the target label of the sample contained in the global label tensor; determine the local maximum value and local exponent sum corresponding to the sample based on the multiple prediction scores corresponding to the sample; if the target label of the sample belongs to the slice range responsible for by the first computing device, then obtain the prediction score corresponding to the target label of the sample from the multiple prediction scores corresponding to the sample.

[0218] In some embodiments, the local statistics module 420 is further configured to: traverse multiple predicted scores corresponding to a sample; if the currently accessed predicted score and the current local maximum value satisfy a first preset condition, then multiply the current local index sum based on the currently accessed predicted score and the current local maximum value to obtain an updated local index sum; use the currently accessed predicted score as the updated local maximum value; if the currently accessed predicted score and the current local maximum value satisfy a second preset condition, then accumulate the current local index sum based on the currently accessed predicted score and the current local maximum value to obtain an updated local index sum; use the current local maximum value as the updated local maximum value; after accessing the last predicted score among the multiple predicted scores corresponding to the sample, determine the local maximum value and the local index sum corresponding to the sample based on the updated local maximum value and the updated local index sum.

[0219] In some embodiments, the local statistics module 420 is further configured to determine the predicted score corresponding to the target label of the sample as a preset value if the target label of the sample does not belong to the segmentation range under the responsibility of the first computing device.

[0220] In some embodiments, the loss calculation module 440 is further configured to: determine the global maximum value and the global exponential sum based on global statistical information; and for each sample, determine the loss value corresponding to the sample based on the predicted score corresponding to the target label of the sample, the global maximum value, and the global exponential sum.

[0221] In some embodiments, the global statistical information includes multiple local statistical information corresponding to the sample. The loss calculation module 440 is further configured to: for each sample, reduce the multiple local maximum values ​​corresponding to the sample according to the global statistical information to obtain the global maximum value corresponding to the sample; correct the multiple local exponential sums corresponding to the sample based on the global maximum value; and reduce the corrected multiple local exponential sums to obtain the global exponential sum corresponding to the sample.

[0222] In some embodiments, the distributed training device further includes a gradient calculation module (not shown in the figure).

[0223] The gradient calculation module is used to determine the gradient of multiple predicted scores corresponding to a sample based on global statistical information. The gradient is used to reflect the degree of influence of changes in predicted scores on the loss value.

[0224] In some embodiments, the global statistics include the global maximum value and the global exponential sum. The gradient calculation module is further configured to: for each of the multiple predicted scores corresponding to a sample, determine the gradient corresponding to the predicted score based on the global maximum value and the global exponential sum; if the target label of the sample belongs to the segmentation range handled by the first computing device, then correct the gradient of the predicted score corresponding to the target label of the sample.

[0225] In some embodiments, the distributed training device further includes a gradient storage module (not shown in the figure).

[0226] The gradient storage module is used to store the gradients of multiple predicted scores corresponding to a sample into the storage space corresponding to the prediction tensor slice.

[0227] The distributed training apparatus provided in this application generates local statistical information, including local maxima, local exponents, and the predicted score corresponding to the target label of a sample, after receiving input data containing prediction tensor fragments and a global label tensor. It then obtains global statistical information by exchanging local statistical information among various computing devices, and finally generates the loss value corresponding to each sample based on the global statistical information. This technical solution extracts prediction tensor fragments, whose data volume is related to the size of the local vocabulary, into local statistical information that is only related to the number of samples. Each sample corresponds to a set of statistical information consisting of local maxima, local exponents, and the predicted score corresponding to the target label, and its data size is much smaller than that of the prediction tensor fragments. Furthermore, since subsequent cross-device communication and loss calculation are based on the aforementioned local statistical information, there is no need to generate and store intermediate tensors with the same data size as the global prediction tensor, thus avoiding excessive storage space consumption caused by storing intermediate tensors. As the model size and / or data size increases, the distributed training apparatus provided in this application can still maintain a low storage space usage, and the training task will not be limited by insufficient storage space, thereby effectively improving the efficiency of distributed training.

[0228] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0229] For example, please refer to Figure 5 , Figure 5 This is a structural block diagram of a computer device provided in one possible implementation of this application. The computer device 500 can be any electronic device with data computing, processing, and storage functions. The computer device 500 can be used to implement the distributed training method provided in the above embodiments.

[0230] Typically, computer device 500 may include a processor 510 and a memory 520.

[0231] Processor 510 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 510 may be implemented using at least one hardware form selected from DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 510 may also include a main processor and a coprocessor. The main processor, also known as the CPU, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 510 may integrate a GPU, which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 510 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0232] Memory 520 may include one or more computer-readable storage media, which may be non-transitory. Memory 520 may also include high-speed random access memory and NVM (Non-Virtual Machine). Volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in memory 520 is used to store a computer program configured to be executed by one or more processors to implement the distributed training described above.

[0233] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the computer device 500, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0234] In an illustrative embodiment, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor of a computer device, implements the distributed training method described above. Optionally, the computer-readable storage medium may be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc.

[0235] In an exemplary embodiment, a chip is also provided, the chip including programmable logic circuitry and / or program instructions stored in a computer-readable storage medium. A processor of a computer device reads the programmable logic circuitry and / or program instructions from the computer-readable storage medium, and executes the programmable logic circuitry and / or program instructions, causing the computer device to perform the distributed training method described above.

[0236] In an exemplary embodiment, a computer program product is also provided, which includes a computer program that is loaded and executed by a processor to implement the distributed training method described above.

[0237] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0238] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A distributed training method, characterized in that, A first computing device is applied in a distributed training system, the distributed training system including the first computing device and at least one other computing device, the method comprising: Receive input data, which includes prediction tensor slices and a global label tensor, wherein the global label tensor includes target labels corresponding to multiple samples respectively; Based on the prediction tensor slices and the global label tensor, at least one sample's local statistical information is generated; wherein, the sample's local statistical information includes: local maximum, local exponent sum, and the prediction score corresponding to the sample's target label; The system sends first local statistical information to each of the other computing devices and receives second local statistical information from the other computing devices to obtain global statistical information; wherein, the first local statistical information includes local statistical information corresponding to the at least one sample generated by the first computing device, and the second local statistical information includes local statistical information corresponding to the at least one sample generated by the other computing devices. Based on the global statistical information, a loss value is generated for each of the at least one sample.

2. The method according to claim 1, characterized in that, The step of generating local statistical information corresponding to at least one sample based on the predicted tensor fragmentation and the global label tensor includes: For each sample, read the multiple prediction scores corresponding to the sample contained in the prediction tensor slice, and the target label of the sample contained in the global label tensor; Based on the multiple predicted scores corresponding to the sample, determine the local maximum and local exponent sum corresponding to the sample; If the target label of the sample belongs to the segmentation range handled by the first computing device, then the prediction score corresponding to the target label of the sample is obtained from the multiple prediction scores corresponding to the sample.

3. The method according to claim 2, characterized in that, The step of determining the local maximum and local exponent sum corresponding to the sample based on the multiple predicted scores corresponding to the sample includes: Iterate through the multiple predicted scores corresponding to the sample; If the predicted score of the current visit and the current local maximum satisfy the first preset condition, then based on the predicted score of the current visit and the current local maximum, the current local exponent sum is multiplied to obtain the updated local exponent sum; the predicted score of the current visit is used as the updated local maximum. If the predicted score of the current visit and the current local maximum value satisfy the second preset condition, then based on the predicted score of the current visit and the current local maximum value, the current local exponent sum is accumulated to obtain the updated local exponent sum; and the current local maximum value is used as the updated local maximum value. After accessing the last predicted score among the multiple predicted scores corresponding to the sample, the local maximum and local exponent sum corresponding to the sample are determined based on the updated local maximum and the updated local exponent sum.

4. The method according to claim 2, characterized in that, The method further includes: If the target label of the sample does not belong to the segmentation range handled by the first computing device, the prediction score corresponding to the target label of the sample is determined to be a preset value.

5. The method according to claim 1, characterized in that, The step of generating loss values ​​for each of the at least one sample based on the global statistical information includes: Based on the global statistics, determine the global maximum value and the global exponent sum; For each sample, the loss value corresponding to the sample is determined based on the predicted score corresponding to the target label of the sample, the global maximum value, and the global exponent sum.

6. The method according to claim 5, characterized in that, The global statistical information includes multiple local statistical information corresponding to the sample; The step of determining the global maximum value and the global exponent sum based on the global statistical information includes: For each sample, based on the global statistical information, the multiple local maxima corresponding to the sample are reduced to obtain the global maximum value corresponding to the sample; Based on the global maximum value, the sum of multiple local indices corresponding to the sample is corrected; the corrected sum of multiple local indices is reduced to obtain the sum of global indices corresponding to the sample.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Based on the global statistical information, the gradients of multiple predicted scores corresponding to the sample are determined, and the gradients are used to reflect the degree of influence of changes in the predicted scores on the loss value.

8. The method according to claim 7, characterized in that, The global statistics include the global maximum value and the global exponent sum; Determining the gradient of multiple predicted scores corresponding to the sample based on the global statistical information includes: For each of the multiple predicted scores corresponding to the sample, the gradient corresponding to the predicted score is determined based on the global maximum value and the global exponent sum; If the target label of the sample belongs to the segmentation range handled by the first computing device, then the gradient of the predicted score corresponding to the target label of the sample is corrected.

9. The method according to claim 7, characterized in that, The method further includes: The gradients of the multiple predicted scores corresponding to the sample are stored in the storage space corresponding to the prediction tensor slice.

10. A distributed training device, characterized in that, A first computing device applied in a distributed training system, the distributed training system including the first computing device and at least one other computing device, the device comprising: The receiving module is used to receive input data, which includes prediction tensor slices and global label tensors, wherein the global label tensor includes target labels corresponding to multiple samples respectively; The local statistics module is used to generate local statistical information corresponding to at least one sample based on the prediction tensor slices and the global label tensor; wherein, the local statistical information corresponding to the sample includes: local maximum value, local index sum, and the prediction score corresponding to the target label of the sample; A global statistics module is used to send first local statistical information to each of the other computing devices and receive second local statistical information from the other computing devices to obtain global statistical information; wherein, the first local statistical information includes local statistical information corresponding to the at least one sample generated by the first computing device, and the second local statistical information includes local statistical information corresponding to the at least one sample generated by the other computing devices. The loss calculation module is used to generate loss values ​​corresponding to the at least one sample based on the global statistical information.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 8.

13. A chip, characterized in that, The chip includes programmable logic circuitry and / or program instructions, which, when the chip is running, are used to implement the method as described in any one of claims 1 to 8.

14. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the method as described in any one of claims 1 to 8.

Images