Data processing method, device, equipment and storage medium

Abstract

The present disclosure provides a data processing method, apparatus, device, and storage medium, which relate to the field of data processing technology, and in particular to computing fields such as artificial intelligence, deep learning, and large models. A specific implementation scheme is as follows: using a first computing unit of a first GPU to obtain a first cross entropy corresponding to a first implicit state feature; the first GPU is one of multiple GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of multiple sub-features obtained after segmenting the target implicit state feature; using the first GPU to obtain a second implicit state feature broadcasted by a second GPU; the second implicit state feature is one of the multiple sub-features; using a second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature; using the first GPU, and based at least on the first cross entropy and the second cross entropy, obtaining a parallel processing result corresponding to the target implicit state feature.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to technical fields such as artificial intelligence, deep learning, and large models. Background Art

[0002] Although Large Language Models (LLMs) have achieved significant breakthroughs in various fields, they can consume excessive video memory and even cause out-of-memory (OOM) problems when processing long sequences in distributed scenarios. Summary of the Invention

[0003] The present disclosure provides a data processing method, apparatus, device, and storage medium.

[0004] According to one aspect of the present disclosure, there is provided a data processing method, comprising:

[0005] Obtaining, using a first computing unit of a first GPU, a first cross entropy corresponding to a first implicit state feature; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained by segmenting the target implicit state feature; and the target implicit state feature is obtained by feature processing text data or image data;

[0006] Obtaining, using the first GPU, a second implicit state feature broadcasted by a second GPU among the plurality of GPUs, wherein the second implicit state feature is one of the plurality of sub-features;

[0007] Obtaining, using a second computing unit of the first GPU, a second cross entropy corresponding to the second implicit state feature;

[0008] A parallel processing result corresponding to a target implicit state feature is obtained using the first GPU and based on at least the first cross entropy and the second cross entropy.

[0009] According to another aspect of the present disclosure, there is provided a data processing apparatus, comprising:

[0010] a first processing unit, configured to obtain, using a first computing unit of a first GPU, a first cross entropy corresponding to a first implicit state feature; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained by segmenting the target implicit state feature; and the target implicit state feature is obtained by feature processing text data or image data;

[0011] a communication unit, configured to obtain, using the first GPU, a second implicit state feature broadcasted by a second GPU among the plurality of GPUs; wherein the second implicit state feature is one of the plurality of sub-features;

[0012] A second processing unit is configured to obtain a second cross entropy corresponding to a second implicit state feature using a second computing unit of the first GPU;

[0013] The fusion processing unit is configured to obtain, using the first GPU and at least based on the first cross entropy and the second cross entropy, a parallel processing result corresponding to the target implicit state feature.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0020] In this way, the disclosed solution provides a refined solution for the parallel processing results corresponding to the target implicit state features (i.e., full features) obtained by the first GPU in a parallel scenario. In this solution, there is no need to perform feature splicing between the first implicit state features and the second implicit state features processed in other GPUs. In other words, there is no need to perform the All Gather operation described above to obtain the parallel processing effect based on the full features (i.e., target implicit state features). In this way, the video memory resources required for parallel computing are reduced, thereby effectively avoiding the problem of insufficient memory when calculating the loss value of the target implicit state features due to the All Gather operation. On the basis of effectively ensuring distributed processing, it lays the foundation for further improving the processing efficiency of large models.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0023] Figure 1(a) is a schematic diagram of a large model distributed training method for long text sequences;

[0024] Figure 1(b) is a schematic diagram of a parallel processing method based on a network structure composed of small operators;

[0025] Figure 2 is a schematic flowchart of a data processing method according to an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of a parallel processing method based on a target fusion operator according to an embodiment of the present application;

[0027] FIG4( a ) is a schematic diagram of a scenario in a specific example of a data processing method according to an embodiment of the present application;

[0028] FIG4( b ) is a schematic diagram of a scenario in another specific example of a data processing method according to another embodiment of the present application;

[0029] Figure 5 1 is a schematic diagram of a scenario of parallel processing in a back propagation process according to a data processing method according to an embodiment of the present application;

[0030] Figure 6 This is a schematic flow chart of a data processing method according to an embodiment of the present application. Figure 2 ;

[0031] FIG7( a ) is a schematic diagram of the structure of a first computing unit in a target fusion operator according to an embodiment of the present application;

[0032] FIG7( b ) is a schematic structural diagram of a first computing unit in a target fusion operator according to another embodiment of the present application;

[0033] Figure 8 This is a schematic flow chart of a data processing method according to an embodiment of the present application. Figure 3 ;

[0034] FIG9( a ) is a schematic diagram of the structure of a second computing unit in a target fusion operator according to an embodiment of the present application;

[0035] FIG9( b ) is a schematic structural diagram of a second computing unit in a target fusion operator according to another embodiment of the present application;

[0036] FIG10( a ) is a schematic diagram of a parallel processing scenario in an example of a forward propagation process of a data processing method according to an embodiment of the present application;

[0037] FIG10( b ) is a schematic diagram of a parallel processing scenario in an example of a forward propagation process of a data processing method according to another embodiment of the present application;

[0038] FIG11( a ) is a schematic diagram of a parallel processing scenario in an example of a back-propagation process according to a data processing method according to an embodiment of the present application;

[0039] FIG11( b ) is a schematic diagram of a parallel processing scenario in an example of a back-propagation process according to a data processing method according to another embodiment of the present application;

[0040] Figure 12 is a structural diagram of a data processing device according to an embodiment of the present application;

[0041] Figure 13 It is a block diagram of an electronic device used to implement the data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0043] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them, and do not mean to limit the order or to limit to only two. For example, the first feature and the second feature refer to two categories / two features. The first feature can be one or more, and the second feature can also be one or more.

[0044] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0045] The following describes the related technologies of the embodiments of the present disclosure. The following related technologies are optional solutions that can be combined with the technical solutions of the embodiments of the present disclosure in any way, and all of them fall within the protection scope of the embodiments of the present disclosure.

[0046] In large-scale model scenarios involving long text sequences, distributed processing can be performed. For example, in a distributed scenario involving collective communication on a large cluster, as shown in Figure 1(a), in a transformer model application scenario using tensor parallelism (TP) and sequence parallelism (SP), for example, in a model training scenario, an accelerator card (e.g., a GPU) typically requires an all-gather operation to collect the hidden state features (e.g., hidden state features with a tensor dimension of [B, S / tp, H]) learned on other accelerator cards (e.g., GPUs) to obtain a complete hidden state (e.g., hidden state features with a tensor dimension of [B, S, H]). Here, B represents the batch size, i.e., the number of sequences processed in a single training iteration; S represents the sequence dimension, i.e., the length of the sequence; H represents the length of the hidden state features; and tp represents the tensor parallel dimension, i.e., the number of hidden state features obtained after segmentation.

[0047] Furthermore, the parallel computing head (also called the parallel head) module is used to project the full hidden state into the output space and calculate the logarithm value (Logits) of the tensor dimension [B, S, V / tp]. Then, the parallel cross entropy module is used to calculate the cross entropy loss value (loss) of the tensor dimension [B, S, 1] based on the Logits, facilitating subsequent model optimization based on the loss. Here, V represents the vocabulary dimension, that is, the number of words or characters in the vocabulary.

[0048] It should be noted that as sequence length increases, for example, to 128KB or even larger, the full amount of Hidden State collected through AllGather will occupy a large amount of video memory space, and may even cause out-of-memory (OOM) problems when calculating logits and loss. For example, in the Large Language Model Meta Artificial Intelligence (LLAMA) 3.2, the vocabulary dimension (V) is 128KB, the length of the implicit state feature (H) is 8192, and the tensor parallel dimension (tp) is 8. At this time, if the 16-bit Brain Floating Point 16 (BF16) data format is used, the sequence length (S) of the vocabulary is 1024KB, the batch size (B) is 1, and the video memory occupied by Logits is 32GB.

[0049] Therefore, to solve the above problem, in one example, a network structure composed of small operators can be used to optimize the memory issues caused by parallel computing. For example, as shown in Figure 1(b), due to the use of tensor parallelism and sequence parallelism, the Hidden State needs to be split along the sequence dimension S. The tensor dimensions of each sub-feature obtained after the split are [S / tp, H]. Furthermore, through the All Gather operation, the full amount of Hidden State (with a tensor dimension of [S, H]) is collected on each accelerator card (for example, GPU 0, GPU 1, etc.). Here, in one example, the Parallel Head module can use the column-slicing method to split the weight (Weight, abbreviated as W) from the vocabulary dimension. For example, the Weight with the tensor [H, V] is split into the Weight with the tensor dimension [H, V / tp]. At this time, the Parallel Head module can be used on each accelerator card to perform matrix product processing on the Hidden State with the tensor dimension [S, H] and the Weight with the tensor dimension [H, V / tp] to obtain Logits [S, V / tp] in the vocabulary dimension segmentation state, and then the Parallel Cross Entropy module is used to perform parallel loss calculation based on Logits [S, V / tp] and Label [S, V] to obtain loss [S, 1].

[0050] In the above scheme, although the network structure composed of small operators can alleviate the problem of insufficient video memory to a certain extent, there are still problems such as unstable video memory control and low computing efficiency, difficult optimization, and difficulty in ensuring the accuracy of the trained model.

[0051] Based on this, the disclosed solution provides a tensor parallel and sequence parallel processing solution. Compared with the existing solution, the disclosed solution can complete the loss calculation of long sequences without using the All Gather operation, which greatly improves the computational efficiency of parallel computing. At the same time, it also reduces the occupied video memory resources, thereby effectively solving the problem of insufficient memory when calculating losses.

[0052] Specifically, Figure 2 FIG1 is a schematic flow chart of a data processing method according to an embodiment of the present application. The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0053] Furthermore, the method includes at least part of the following contents. Figure 2 As shown, including:

[0054] Step S201: using a first computing unit of a first GPU to obtain a first cross entropy corresponding to a first implicit state feature.

[0055] Here, the first GPU is one of the multiple GPUs required for parallel processing of the target implicit state feature; the first implicit state feature is one of the multiple sub-features obtained after splitting the target implicit state feature (also known as Hidden State) (for example, splitting according to the sequence dimension).

[0056] Furthermore, in one example, the target implicit state feature is obtained by performing feature processing on text data or image data.

[0057] For example, in one example, feature learning is performed on text data or image data to obtain a Hidden State with a tensor dimension of [S, H] (corresponding to the target implicit state feature above). At this time, the Hidden State with a tensor dimension of [S, H] is split on the sequence dimension to obtain multiple sub-features corresponding to the Hidden State. For example, the tensor dimension of each sub-feature can be [S / tp, H], where tp represents the tensor parallel dimension (that is, the number of implicit state features obtained after splitting).

[0058] Step S202: using the first GPU, obtaining a second implicit state feature broadcasted by a second GPU among the multiple GPUs.

[0059] Here, the second implicit state feature is one of the multiple sub-features.

[0060] Furthermore, it is understood that the second implicit state feature is different from the first implicit state feature. This facilitates the use of multiple GPUs to process the target implicit state feature in parallel, thereby improving processing efficiency.

[0061] Step S203: using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

[0062] It should be pointed out that the "cross entropy" referred to in the present disclosure can be specifically understood as the cross entropy loss value (which can be simply referred to as loss), that is, the loss value calculated using the cross entropy loss function.

[0063] Step S204: using the first GPU and obtaining a parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy.

[0064] It should be noted that, in this example, since the parallel processing result is obtained by processing the first implicit state feature and the second implicit state feature using the first GPU, the parallel processing result can be understood as the parallel processing result of the target implicit state feature on the first GPU.

[0065] In this way, the disclosed solution provides a refined solution for the parallel processing results corresponding to the target implicit state features (i.e., full features) obtained by the first GPU in a parallel scenario. In this solution, there is no need to perform feature splicing between the first implicit state features and the second implicit state features processed in other GPUs. In other words, there is no need to perform the All Gather operation described above to obtain the parallel processing effect based on the full features (i.e., target implicit state features). In this way, the video memory resources required for parallel computing are reduced, thereby effectively avoiding the problem of insufficient memory when calculating the loss value of the target implicit state features due to the All Gather operation. On the basis of effectively ensuring distributed processing, it lays the foundation for further improving the processing efficiency of large models.

[0066] Furthermore, since the disclosed solution can utilize multiple computing units to perform parallel calculations on the implicit state features obtained after segmentation, compared with the parallel computing solution in the prior art that uses the All Gather operation to collect the full amount of implicit state features, the disclosed solution simplifies the parallel computing process, reduces the overall complexity of parallel computing and the difficulty of optimization, and, while effectively ensuring the efficiency of parallel computing, effectively reduces the peak value of the video memory of the first GPU, providing strong support for improving the model accuracy of large models.

[0067] For example, GPU0 and GPU1 are used to process the target implicit state feature in parallel. At this time, the target implicit state feature can be divided into two parts in the sequence dimension, namely the first implicit state feature and the second implicit state feature. At this time, Figure 3As shown, the disclosed solution can directly input each implicit state feature (such as the first implicit state feature and the second implicit state feature) in the sequence dimension segmentation state into the computing unit (also called the target fusion operator, for example, it can be specifically the fused parallel cross entropy FusedHeadParallelCrossEntropy) of the corresponding GPU (such as GPU0 or GPU1). In this way, the parallel processing results of the target implicit state feature (that is, the full feature) on each GPU are obtained. Compared with the parallel processing solution shown in Figure 1(b), the parallel processing efficiency of the disclosed solution is higher, which reduces the overall complexity and optimization difficulty of parallel computing. At the same time, while effectively ensuring the parallel computing efficiency, it also effectively reduces the peak value of the video memory of each GPU.

[0068] Further, continue with Figure 3 Taking the example shown in FIG4(a), as shown in FIG4(a), the first computing unit in GPU 0 (corresponding to the first GPU above) is used to obtain the first cross entropy corresponding to the sub-feature X0 (corresponding to the first implicit state feature above) of the target implicit state feature; and GPU 0 is used to obtain the sub-feature X1 (corresponding to the second implicit state feature above) of the target implicit state feature sent by GPU 1 (corresponding to the second GPU above) through broadcast communication, and the second computing unit in GPU 0 is used to obtain the second cross entropy corresponding to the sub-feature X1; further, the aggregation unit in GPU 0 is used to aggregate the obtained first cross entropy and the second cross entropy to obtain the parallel processing result corresponding to the target implicit state feature. In this way, the video memory resources required for parallel computing are effectively reduced, laying the foundation for subsequently improving the processing efficiency of large models.

[0069] It should be noted that the number of GPUs required for parallel processing of the target implicit state feature is related to the parallel dimension of the tensor (i.e., tp). In other words, the number of GPUs required for parallel processing is related to the number of sub-features obtained after the target implicit state feature is divided. For example, in one example, if the number of GPUs required for parallel computing is N, the target implicit state feature can be divided and N sub-features can be obtained. At this time, the above-mentioned data processing method can also specifically include: using the first computing unit of the first GPU to obtain a first cross entropy corresponding to the first implicit state feature; using the first GPU to obtain the second implicit state feature broadcast by each second GPU in N-1 second GPUs; using the N-1 second computing units of the first GPU to parallel process the N-1 second implicit state features to obtain the second cross entropy corresponding to each second implicit state feature; and then using the first GPU, and based on the first cross entropy and the second cross entropy corresponding to each second implicit state feature, obtain the parallel processing result corresponding to the target implicit state feature. In this way, multiple computing units are used to process each sub-feature in parallel, thereby reducing the peak memory usage of the first GPU and improving the parallel processing efficiency.

[0070] For example, using three GPUs to process the three sub-features obtained after segmentation in parallel, as shown in Figure 4(b), computing unit 1 in GPU 0 is used to obtain the cross-entropy 1 corresponding to sub-feature X0; GPU 0 is used to obtain sub-feature X1 sent by GPU 1 via broadcast communication, and computing unit 2 in GPU 0 is used to obtain the cross-entropy 2 corresponding to sub-feature X1; GPU 0 is used to obtain sub-feature X2 sent by GPU 2 via broadcast communication, and computing unit 3 in GPU 0 is used to obtain the cross-entropy 3 corresponding to sub-feature X2; further, the aggregation unit in GPU 0 is used to aggregate the first cross-entropy corresponding to sub-feature X0, the second cross-entropy corresponding to sub-feature X1, and the second cross-entropy corresponding to sub-feature X2 to obtain the parallel processing result corresponding to the target implicit state feature. In this way, the peak memory usage of the first GPU during the parallel computing process is effectively reduced, laying the foundation for subsequent improvements in the training efficiency of large models.

[0071] It should be noted that the above is only an example. In actual applications, the number of sub-features obtained after segmenting the target implicit state features and the number of GPUs may be one-to-one corresponding or not. They can be set according to actual conditions, and the present disclosure does not impose specific restrictions on this.

[0072] Furthermore, in a specific example, the second implicit state feature is located after the first implicit state feature in the sequence dimension of the target implicit state feature. This makes it easier to utilize multiple GPUs for parallel processing in the sequence dimension, thereby effectively achieving sequence parallelism.

[0073] Furthermore, in a specific example, the above-described method of using the first GPU and obtaining the parallel processing result corresponding to the target implicit state feature based on at least the first cross entropy and the second cross entropy (e.g., step S204) may specifically include:

[0074] In the forward propagation process, the total cross entropy corresponding to the target implicit state feature is obtained through the first GPU based on the first cross entropy and the second cross entropy.

[0075] That is to say, in one example, in the forward propagation process, the parallel processing result obtained using the above steps S201 to S204 can be specifically the total cross entropy corresponding to the target implicit state feature on the first GPU. The specific process details can be referred to the example of Figure 4(a), which will not be repeated here.

[0076] In this way, the disclosed solution quickly obtains the total cross entropy in forward propagation through tensor parallel and sequence parallel processing solutions. The above process effectively reduces the GPU memory peak during parallel computing, thus laying the foundation for subsequent improvement of the processing efficiency of large models.

[0077] Alternatively, in another specific example, the above-described method of using the first GPU and obtaining the parallel processing result corresponding to the target implicit state feature based on at least the first cross entropy and the second cross entropy (e.g., step S204) may specifically include:

[0078] In the back propagation process, the first GPU is used to obtain a first reverse gradient corresponding to the first cross entropy and a second reverse gradient corresponding to the second cross entropy.

[0079] That is, in one example, in the back propagation process, the parallel processing results obtained by using the above steps S201 to S204 can be specifically the first reverse gradient corresponding to the first cross entropy and the second reverse gradient of the second cross entropy. Figure 3 For example, Figure 5As shown, in the back propagation process, the recalculation (Recompute) technology is used, and the first cross entropy is obtained by the first computing unit in GPU0. Here, the step of obtaining the first cross entropy in the back propagation process is similar to the step of obtaining the first cross entropy in the forward propagation process, and will not be repeated here. Further, after obtaining the first cross entropy by the first computing unit in GPU0, the first gradient calculation unit is used, and according to the relevant data required to calculate the first reverse gradient (such as the first relevant data), the first reverse gradient corresponding to the first cross entropy is obtained, that is, the first reverse gradient for the sub-feature X0 is obtained, for example, including the reverse gradient X0 and the reverse gradient W0.

[0080] Similarly, the recalculation (Recompute) technology is used, and the second cross entropy is obtained by the second computing unit in GPU0. Here, the step of obtaining the second cross entropy in the back propagation process is similar to the step of obtaining the second cross entropy in the forward propagation process, and will not be repeated here. Further, after obtaining the second cross entropy by the second computing unit in GPU0, the second gradient calculation unit is used, and based on the relevant data required for calculating the second reverse gradient (such as the second relevant data), the second reverse gradient corresponding to the second cross entropy is obtained, that is, the second reverse gradient for the sub-feature X1 is obtained, for example, including the reverse gradient X1 and the reverse gradient W0.

[0081] In this way, the disclosed solution can recompute the technology to quickly obtain the reverse gradient during back propagation. Moreover, the above process effectively reduces the peak value of GPU memory during parallel computing, thus laying the foundation for subsequent improvement of the processing efficiency of large models.

[0082] Figure 6 This is a schematic flow chart of a data processing method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters and other electronic devices. It is understood that the above Figures 2 to 5 The relevant contents of the method shown can also be applied to this example, and this example will not elaborate on the relevant contents.

[0083] Furthermore, the method includes at least part of the following contents. Figure 6 As shown, including:

[0084] Step S601: Utilize the first feature projection module in the first computing unit of the first GPU to process the first implicit state feature to obtain a first initial prediction value.

[0085] Here, the first GPU is one of the multiple GPUs required for parallel processing of the target implicit state feature; the first implicit state feature is one of the multiple sub-features obtained after splitting the target implicit state feature (also known as Hidden State) (for example, splitting according to the sequence dimension).

[0086] Furthermore, in one example, the target implicit state feature is obtained by performing feature processing on text data or image data.

[0087] It should be noted that the number of GPUs and related sub-features can be found in the above examples and will not be elaborated here.

[0088] Furthermore, the first initial prediction value may specifically be the first logical logarithm value output by the last layer in the model, for example, the logits value that has not been normalized.

[0089] Step S602: Utilize the first loss calculation module in the first calculation unit to process the first initial prediction value to obtain a first cross entropy corresponding to the first implicit state feature.

[0090] That is, in the disclosed solution, first, the first implicit state feature is processed using the first feature projection module in the first computing unit in the first GPU (for example, the first implicit state feature is multiplied by the weight in the first GPU) to obtain a first initial prediction value, and then the first loss calculation module in the first computing unit is used to obtain the first cross entropy based on the obtained first initial prediction value. In this way, it is convenient to obtain the parallel processing result corresponding to the target implicit state feature based on the first cross entropy and the second cross entropy.

[0091] For example, continue with Figure 3 Taking the example shown in FIG7 (a) as an example, the first computing unit in GPU 0 (corresponding to the first GPU above) is used to process the sub-feature X0 of the target implicit state feature. This may specifically include: first using the first feature projection module in the first computing unit to process the sub-feature X0 to obtain the first Logits (corresponding to the first initial prediction value above), and then using the first loss calculation module in the first computing unit (such as the cross entropy loss calculation module), and according to the obtained first Logits, calculating the first cross entropy corresponding to the sub-feature X0. In this way, data support is provided for the subsequent parallel processing results corresponding to the target implicit state feature on the first GPU.

[0092] Step S603: using the first GPU, obtaining a second implicit state feature broadcasted by a second GPU among the multiple GPUs.

[0093] Here, the second implicit state feature is one of the multiple sub-features.

[0094] It should be noted that, for the relevant content about the second implicit state feature, please refer to the above example and will not be elaborated here.

[0095] Step S604: using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

[0096] Step S605: using the first GPU and obtaining a parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy.

[0097] In this way, the disclosed solution provides a refined solution for using the first computing unit to process the first implicit state feature, that is, using multiple modules in the first computing unit to process the first implicit state feature. In this way, the peak value of the video memory of the first GPU during the calculation process is effectively reduced, and the efficiency of parallel computing is further improved, thereby laying the foundation for the subsequent training and optimization of large models.

[0098] Furthermore, in a specific example, the first cross entropy can be obtained in the following manner; specifically, the above-described processing of the first initial prediction value using the first loss calculation module in the first calculation unit to obtain the first cross entropy corresponding to the first implicit state feature (for example, step S602) can specifically include:

[0099] Step S602-1: Utilize the first loss calculation module in the first computing unit to obtain first feature data required for calculating the first cross entropy from at least the second GPU.

[0100] Step S602-2: Utilizing the first loss calculation module in the first computing unit and based at least on the first feature data obtained from the second GPU, calculate a first cross entropy corresponding to the first implicit state feature.

[0101] That is to say, in the process of processing the first initial prediction value using the first loss calculation module in the first computing unit, it is also necessary to use the first loss calculation module to obtain the first feature data required for calculating the first cross entropy from the second GPU, and then the first loss calculation module obtains the first cross entropy based on the obtained first feature data and the first initial prediction value.

[0102] It should be noted that when the number of sub-features obtained by segmenting the target implicit state feature is N (N is greater than 2), the first feature data required for calculating the first cross entropy may be N-1. In this case, the first loss calculation module in the first computing unit can be used to obtain the first feature data required for calculating the first cross entropy from each of the N-1 second GPUs, and then the first loss calculation module in the first computing unit can be used to calculate the first cross entropy corresponding to the first implicit state feature based on the first feature data obtained from each second GPU. In this way, the accuracy and reliability of the subsequent parallel processing results are ensured.

[0103] In this way, the disclosed solution further provides a refined solution for obtaining the first cross entropy. The solution is simple, efficient, and can quickly calculate the first cross entropy. Moreover, the feature data used in the above process does not need to be spliced ​​with the features of other second GPUs. Therefore, compared with the existing solution, the peak value of the video memory of the first GPU during the calculation process is effectively reduced, thereby effectively ensuring efficient parallel computing and laying the foundation for subsequent training and optimization of large models.

[0104] In a specific example, in order to further reduce the peak value of video memory during the calculation process, the first implicit state may be divided into blocks, the divided data blocks may be processed in parallel, and the processing results corresponding to each data block may be fused to obtain the first cross entropy. Specifically, the above-mentioned use of the first computing unit of the first GPU to obtain the first cross entropy corresponding to the first implicit state feature (for example, step S201) may specifically include:

[0105] Step a: Using the first computing unit of the first GPU, split the first implicit state feature to obtain at least two first data blocks.

[0106] Step b: using the first computing unit of the first GPU to perform parallel processing on the at least two first data blocks to obtain a sub-cross entropy corresponding to each first data block.

[0107] Step c: using the first computing unit of the first GPU, fusing the sub-cross entropies corresponding to each first data block to obtain a first cross entropy corresponding to the first implicit state feature.

[0108] In this way, the disclosed solution first divides the first implicit state feature, and then processes the divided data blocks in parallel to obtain sub-cross entropies of each data block in parallel, and then fuses the sub-cross entropies of each data block to obtain the first cross entropy. In this way, the disclosed solution uses subbatch technology to further reduce the peak value of the video memory of the first GPU during the intermediate calculation process, thereby further saving the required resources and time. At the same time, it also further improves the parallel computing efficiency, laying the foundation for subsequent obtaining the parallel processing results corresponding to the target implicit state features and improving the training efficiency of the model.

[0109] It should be noted that the above-mentioned process of "processing the divided data blocks in parallel" can also be applied to the detailed process of step S601 to step S602. For example, after the first implicit state feature is divided to obtain M (an integer greater than or equal to 2) first data blocks, the M first feature projection sub-modules in the first calculation unit are used to process the M first data blocks in parallel to obtain the first initial prediction sub-value of each first data block, and then the M first loss calculation sub-modules in the first calculation unit are used to process the obtained first initial prediction sub-value of each first data block in parallel to obtain the sub-cross entropy of each first data block; finally, the aggregation module in the first calculation unit is used to aggregate the sub-cross entropy of each first data block to obtain the first cross entropy corresponding to the first implicit state feature.

[0110] For example, continue with Figure 3 Taking the example shown in FIG7(b), the first computing unit in GPU 0 (corresponding to the first GPU above) is used to split the sub-feature X0 into data blocks X00 and X01. The first feature projection submodule 1 in the first computing unit is used to obtain the first Logits subvalue 1 of the data block X00. The first loss calculation submodule 1 in the first computing unit is then used to process the first Logits subvalue 1 to obtain the sub-cross entropy (i.e., sub-cross entropy 01) of the data block X00. Correspondingly, the first feature projection submodule 2 in the first computing unit is used to obtain the first Logits subvalue 2 of the data block X01. The first loss calculation submodule 2 in the first computing unit is then used to process the first Logits subvalue 2 to obtain the sub-cross entropy (i.e., sub-cross entropy 02) of the data block X01. The aggregation module in the first computing unit is then used to aggregate the sub-cross entropies of each data block to obtain the first cross entropy corresponding to the first implicit state feature. In this way, the peak memory usage of the first GPU during the calculation process is further reduced.

[0111] Figure 8 This is a schematic flow chart of a data processing method according to an embodiment of the present application. Figure 3The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters and other electronic devices. It is understood that the above Figure 2 The relevant contents of the method shown in FIG7 can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0112] Furthermore, the method includes at least part of the following contents. Figure 8 As shown, including:

[0113] Step S801: using a first computing unit of a first GPU to obtain a first cross entropy corresponding to a first implicit state feature.

[0114] Here, the first GPU is one of the multiple GPUs required for parallel processing of the target implicit state feature; the first implicit state feature is one of the multiple sub-features obtained after splitting the target implicit state feature (also known as Hidden State) (for example, splitting according to the sequence dimension).

[0115] Furthermore, in one example, the target implicit state feature is obtained by performing feature processing on text data or image data.

[0116] It should be noted that the number of GPUs and related sub-features can be found in the above examples and will not be elaborated here.

[0117] Furthermore, the first initial prediction value may specifically be the first logical logarithm value output by the last layer in the model, for example, the logits value that has not been normalized.

[0118] Step S802: using the first GPU, obtaining a second implicit state feature broadcasted by a second GPU among the multiple GPUs.

[0119] Here, the second implicit state feature is one of the multiple sub-features.

[0120] It should be noted that, for the relevant content about the second implicit state feature, please refer to the above example and will not be elaborated here.

[0121] Step S803: Utilize the second feature projection module in the second computing unit of the first GPU to process the second implicit state feature to obtain a second initial prediction value.

[0122] Furthermore, the second initial prediction value may specifically be the second logistic value output by the last layer in the model, for example, the logits value that has not been normalized.

[0123] Step S804: Utilize the second loss calculation module in the second calculation unit to process the second initial prediction value to obtain a second cross entropy corresponding to the second implicit state feature.

[0124] That is, in the disclosed solution, first, the second implicit state feature is processed using the second feature projection module in the second computing unit in the first GPU (for example, the second implicit state feature is multiplied by the weight in the first GPU) to obtain a second initial prediction value, and then the second loss calculation module in the second computing unit is used to obtain the second cross entropy based on the obtained second initial prediction value. In this way, it is convenient to obtain the parallel processing result corresponding to the target implicit state feature based on the first cross entropy and the second cross entropy.

[0125] For example, continue with Figure 3 Taking the example shown in FIG9(a) as an example, the second computing unit in GPU 0 (corresponding to the first GPU above) is used to process the sub-feature X1 of the target implicit state feature. This may specifically include: first using the second feature projection module in the second computing unit to process the sub-feature X1 to obtain the second Logits (corresponding to the second initial prediction value above), and then using the second loss calculation module in the second computing unit (such as the cross entropy loss calculation module), and calculating the second cross entropy corresponding to the sub-feature X1 based on the obtained second Logits. In this way, data support is provided for the subsequent parallel processing results corresponding to the target implicit state feature on the first GPU.

[0126] Step S805: using the first GPU and obtaining a parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy.

[0127] In this way, the disclosed solution provides a refined solution for using the second computing unit to process the second implicit state features, that is, using multiple modules in the second computing unit to process the second implicit state features. In this way, the peak value of the video memory of the first GPU during the calculation process is effectively reduced, and the efficiency of parallel computing is further improved, thereby laying the foundation for the subsequent training and optimization of large models.

[0128] Furthermore, in a specific example, the second cross entropy can be obtained in the following manner; specifically, the above-described processing of the second initial prediction value using the second loss calculation module in the second calculation unit to obtain the second cross entropy corresponding to the second implicit state feature (for example, step S804) can specifically include:

[0129] Step S804-1: Utilize the second loss calculation module in the second computing unit to obtain second feature data required for calculating the second cross entropy from at least the second GPU.

[0130] Step S804-2: Utilizing the second loss calculation module in the second computing unit and based at least on the second feature data obtained from the second GPU, calculate a second cross entropy corresponding to the second implicit state feature.

[0131] That is to say, in the process of processing the second initial prediction value using the second loss calculation module in the second computing unit, it is also necessary to use the second loss calculation module to obtain the second feature data required for calculating the second cross entropy from the second GPU, and then the second loss calculation module obtains the second cross entropy based on the obtained second feature data and the second initial prediction value.

[0132] It should be noted that when the number of sub-features obtained by segmenting the target implicit state feature is N (N is greater than 2), the second feature data required for calculating the second cross entropy may be N-1. In this case, the second loss calculation module in the second computing unit can be used to obtain the second feature data required for calculating the second cross entropy from each of the N-1 second GPUs, and then the second loss calculation module in the second computing unit can be used to calculate the second cross entropy corresponding to the second implicit state feature based on the second feature data obtained from each second GPU. In this way, the accuracy and reliability of the subsequent parallel processing results are ensured.

[0133] In this way, the disclosed solution further provides a refined solution for obtaining the second cross entropy. The solution is simple, efficient, and can quickly calculate the second cross entropy. Moreover, the feature data used in the above process does not need to be spliced ​​with the features of other second GPUs. Therefore, compared with the existing solution, the peak memory usage of the first GPU during the calculation process is effectively reduced, thereby laying the foundation for the subsequent training and optimization of large models on the basis of effectively ensuring efficient parallel computing.

[0134] In a specific example, in order to further reduce the peak value of video memory during the calculation process, the second implicit state can be divided into blocks, the divided data blocks are processed in parallel, and the processing results corresponding to each data block are fused to obtain the second cross entropy; specifically, the above-mentioned use of the second computing unit of the first GPU to obtain the second cross entropy corresponding to the second implicit state feature (for example, step S203) can specifically include:

[0135] Step a: Using the second computing unit of the first GPU, split the second implicit state feature to obtain at least two second data blocks.

[0136] Step b: using the second computing unit of the first GPU to perform parallel processing on the at least two second data blocks to obtain a sub-cross entropy corresponding to each second data block.

[0137] Step c: using the second computing unit of the first GPU, the sub-cross entropies corresponding to each second data block are fused to obtain a second cross entropy corresponding to the second implicit state feature.

[0138] In this way, the disclosed solution first divides the second implicit state feature, and then processes the divided data blocks in parallel to obtain sub-cross entropies of each data block in parallel, and then fuses the sub-cross entropies of each data block to obtain the second cross entropy. In this way, the disclosed solution uses subbatch technology to further reduce the peak value of the video memory of the first GPU during the intermediate calculation process, thereby further saving the required resources and time. At the same time, it also further improves the parallel computing efficiency, laying the foundation for subsequent obtaining the parallel processing results corresponding to the target implicit state feature and improving the training efficiency of the model.

[0139] It should be noted that the above-mentioned process of "processing the divided data blocks in parallel" can also be applied to the refined process of step S803 to step S804. For example, after the second implicit state feature is divided to obtain M (an integer greater than or equal to 2) second data blocks, the M second feature projection sub-modules in the second computing unit are used to process the M second data blocks in parallel to obtain the second initial prediction sub-value of each second data block, and then the M second loss calculation sub-modules in the second computing unit are used to process the obtained second initial prediction sub-value of each second data block in parallel to obtain the sub-cross entropy of each second data block; finally, the aggregation module in the second computing unit is used to aggregate the sub-cross entropy of each second data block to obtain the second cross entropy corresponding to the second implicit state feature.

[0140] For example, continue with Figure 3Taking the example shown in FIG9(b), the second computing unit in GPU 0 (corresponding to the first GPU above) is used to split the sub-feature X1 into data blocks X10 and X11. The second feature projection submodule 1 in the second computing unit is used to obtain the second Logits subvalue 1 of the data block X10. The second loss calculation submodule 1 in the second computing unit is then used to process the second Logits subvalue 1 to obtain the sub-cross entropy (i.e., sub-cross entropy 11) of the data block X10. Correspondingly, the second feature projection submodule 2 in the second computing unit is used to obtain the second Logits subvalue 2 of the data block X11. The second loss calculation submodule 2 in the second computing unit is then used to process the second Logits subvalue 2 to obtain the sub-cross entropy (i.e., sub-cross entropy 12) of the data block X11. The aggregation module in the second computing unit is then used to aggregate the sub-cross entropies of each data block to obtain the second cross entropy corresponding to the second implicit state feature. In this way, the peak memory usage of the first GPU during the calculation process is further reduced.

[0141] It should be noted that the disclosed solution mainly describes the processing flow of the first GPU in detail. In actual scenarios, the processing process of other GPUs (for example, the second GPU) is similar to that of the first GPU and will not be described in detail here.

[0142] The following further describes the disclosed solution in detail with reference to FIG. 10(a), FIG. 10(b) and FIG. 11(a), FIG. 11(b); Figure 3Taking the example shown as an example, GPU0 and GPU1 are used to process the target implicit state features in parallel. At this time, the target implicit state features can be divided into two parts in the sequence dimension, namely the first implicit state features and the second implicit state features. Furthermore, as shown in Figure 10(a), in the forward propagation process, the feature projection module 1 in GPU0 is used to perform matrix product processing on the sub-feature X0 and the weight 0 in GPU 0 (also denoted as W0) to obtain the initial prediction value 1, and then the loss calculation module 1 in GPU0 is used to calculate the cross entropy 1 based on the initial prediction value 1 and the feature data 1 obtained from GPU1. And using the feature projection module 2 in GPU0, the sub-feature X1 broadcast by GPU1 is matrix-producted with the weight 0 in GPU 0 to obtain the initial prediction value 2, and then using the loss calculation module 2 in GPU0, the cross entropy 2 is calculated based on the initial prediction value 2 and the feature data 2 obtained from GPU1; further, in GPU 0, using the aggregation unit 1, the cross entropy 1 and the cross entropy 2 are aggregated to obtain the total cross entropy loss of the target implicit state feature on GPU0 (i.e., loss 1); similarly, the total cross entropy loss of the target implicit state feature on GPU1 (i.e., loss 2) can be obtained. Here, it can be understood that the processing flow in GPU1 is similar to that in GPU0, and will not be repeated here.

[0143] Furthermore, the disclosed solution can also divide the sub-features to obtain multiple data blocks, and then process each data block in parallel to obtain cross entropy 1. Here, the calculation unit 1 is taken as an example for explanation. As shown in Figure 10 (b), the sub-feature X0 is divided into data block X00 and data block X01. At this time, the matrix product submodule 1 in GPU0 is used to perform matrix product processing on the data block X00 and the weight 0 to obtain the initial prediction value 01. Then, the loss calculation submodule 1 in GPU0 is used to obtain the sub-cross entropy based on the initial prediction value 01 and the loss obtained from GPU1. 01 partial feature data 1, obtain the sub-cross entropy 01 corresponding to the data block X00; similarly, using the matrix product submodule 2 in GPU0, the data block X01 and the weight 0 are matrix-producted to obtain the initial prediction value 02, and then using the loss calculation submodule 2 in GPU0, according to the initial prediction value 02 and the partial feature data 1 obtained from GPU1 for obtaining the sub-cross entropy 02, the sub-cross entropy 02 corresponding to the data block X01 is obtained; further, the sub-cross entropies corresponding to each data block are aggregated to obtain the cross entropy 1 corresponding to the sub-feature X0. Here, a similar method can be used to obtain cross entropy 2, cross entropy 3 and cross entropy 4, and then loss1 and loss2, which will not be repeated here.

[0144] In summary, for the forward propagation process, the disclosed solution has the following advantages:

[0145] First, it simplifies the processing flow. Specifically, the disclosed solution removes the All Gather operator and can directly input the sub-features of the target implicit state feature in the sequence dimension segmentation state (for example, HiddenState[S / tp, H]), and then calculates the Logits and cross-entropy loss values ​​in sequence according to the order of each sub-feature on the GPU to obtain the total cross-entropy loss of the target implicit state feature on each GPU. In this way, the calculation process is simplified, the GPU memory peak is effectively reduced during the calculation process, and the efficiency of parallel computing is improved.

[0146] Second, lower video memory. Specifically, the disclosed solution can also use Subbatch technology on the GPU to calculate the cross entropy of sub-features, that is, to split the sub-features of the target implicit state feature again, such as splitting the sub-feature HiddenState[S / tp, H] into data blocks HiddenState[chunk_size, H], and then cyclically calculate the sub-cross entropy of each data block, and finally aggregate the sub-cross entropy of each data block in the sequence dimension to obtain the cross entropy of the sub-feature, thus further reducing the GPU video memory peak during the calculation process.

[0147] For another example, as shown in Figure 11(a), in the back-propagation process, a recompute technique is used, and cross-entropy 1 is obtained through computing unit 1 in GPU0. Here, the steps for obtaining cross-entropy 1 in the back-propagation process are similar to the steps for obtaining cross-entropy 1 in the forward propagation process shown in Figure 10(a), and will not be repeated here. Furthermore, after obtaining cross-entropy 1 through computing unit 1 in GPU0, gradient calculation unit 1 is used, and based on the relevant data required for calculating subsequent reverse gradients (e.g., the first relevant data), the reverse gradient X0-0 and reverse gradient W0-0 for sub-feature X0 are obtained on GPU0. Similarly, the reverse gradient X1-0 and reverse gradient W0-1 for sub-feature X1 can also be obtained on GPU0, and the reverse gradient X0-1 and reverse gradient W1-0 for sub-feature X0, and the reverse gradient X1-1 and reverse gradient W1-1 for sub-feature X1 are obtained on GPU1.

[0148] Furthermore, the disclosed solution can also segment the sub-features to obtain multiple data blocks, and then process each data block in parallel to obtain the reverse gradient for the sub-feature. Here, the calculation unit 1 is used as an example for explanation, as shown in Figure 11(b). Here, the specific process of Figure 11(b) is similar to the processing flow of Figure 10(b) and will not be repeated here.

[0149] In summary, for the backpropagation process, the disclosed solution adopts the Recompute technology to re-execute the forward propagation process and then calculate the reverse gradient. Compared with caching the processing results of the forward propagation, the disclosed solution further reduces the GPU memory peak during the calculation process and also reduces the cache space occupied.

[0150] It should be noted that the disclosed solution does not limit the specific calculation process of obtaining the above-mentioned Logits, cross entropy, and reverse gradient.

[0151] Furthermore, in summary, the disclosed solution has the following advantages:

[0152] First, the disclosed solution uses segmentation and broadcast communication operations in the fusion operator (i.e., FusedHeadParallelCrossEntropy) to broadcast the data information in the current GPU to other GPUs, so that other GPUs can use the acquired data information to calculate matrix products and cross-entropy loss values. Therefore, the disclosed solution can significantly reduce the peak value of video memory.

[0153] Second, the disclosed solution uses Subbatch technology within the fusion operator to further reduce the peak memory usage during the intermediate calculation process.

[0154] Third, during backpropagation, the disclosed solution also utilizes Recompute technology, combined with operator fusion technology and Subbatch technology, to collaboratively achieve a reduction in peak memory usage during training of long sequence models, thereby enabling the disclosed solution to support distributed training of models with more than one trillion parameters in scenarios where the sequence length is greater than 128K.

[0155] The present disclosure also provides a data processing device, such as Figure 12 As shown, including:

[0156] A first processing unit 1201 is configured to obtain a first cross entropy corresponding to a first implicit state feature using a first computing unit of a first GPU; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained by segmenting the target implicit state feature; and the target implicit state feature is obtained by feature processing text data or image data;

[0157] A communication unit 1202 is configured to obtain, using the first GPU, a second implicit state feature broadcasted by a second GPU among the multiple GPUs; wherein the second implicit state feature is one of the multiple sub-features;

[0158] The second processing unit 1203 is configured to obtain a second cross entropy corresponding to the second implicit state feature using the second computing unit of the first GPU;

[0159] The fusion processing unit 1204 is configured to obtain a parallel processing result corresponding to the target implicit state feature by using the first GPU and based on at least the first cross entropy and the second cross entropy.

[0160] In a specific example of the disclosed solution, the first processing unit is specifically configured to:

[0161] Processing the first implicit state feature using a first feature projection module in a first computing unit of the first GPU to obtain a first initial prediction value;

[0162] The first initial prediction value is processed using a first loss calculation module in the first calculation unit to obtain a first cross entropy corresponding to the first implicit state feature.

[0163] In a specific example of the disclosed solution, the first processing unit is specifically configured to:

[0164] Obtaining, using the first loss calculation module in the first computing unit, first feature data required for calculating the first cross entropy from at least the second GPU;

[0165] A first cross entropy corresponding to the first implicit state feature is calculated using the first loss calculation module in the first computing unit and based on at least the first feature data obtained from the second GPU.

[0166] In a specific example of the disclosed solution, the first processing unit is specifically configured to:

[0167] Using a first computing unit of a first GPU, segment the first implicit state feature to obtain at least two first data blocks;

[0168] Using a first computing unit of the first GPU, the at least two first data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each first data block;

[0169] The sub-cross entropies corresponding to each first data block are fused using the first computing unit of the first GPU to obtain a first cross entropy corresponding to the first implicit state feature.

[0170] In a specific example of the disclosed solution, the second processing unit is specifically configured to:

[0171] Processing the second implicit state feature using a second feature projection module in a second computing unit of the first GPU to obtain a second initial prediction value;

[0172] The second initial prediction value is processed using a second loss calculation module in the second calculation unit to obtain a second cross entropy corresponding to the second implicit state feature.

[0173] In a specific example of the disclosed solution, the second processing unit is specifically configured to:

[0174] Obtaining, using a second loss calculation module in the second computing unit, second feature data required for calculating a second cross entropy from at least the second GPU;

[0175] A second cross entropy corresponding to the second implicit state feature is calculated using a second loss calculation module in the second computing unit and based at least on the second feature data obtained from the second GPU.

[0176] In a specific example of the disclosed solution, the second processing unit is specifically configured to:

[0177] Using a second computing unit of the first GPU, segment the second implicit state feature to obtain at least two second data blocks;

[0178] Using a second computing unit of the first GPU, the at least two second data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each second data block;

[0179] The sub-cross entropies corresponding to each second data block are fused using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

[0180] In a specific example of the disclosed solution, the fusion processing unit is specifically configured to:

[0181] In the forward propagation process, the total cross entropy corresponding to the target implicit state feature is obtained through the first GPU based on the first cross entropy and the second cross entropy.

[0182] In a specific example of the disclosed solution, the fusion processing unit is specifically configured to:

[0183] In the back propagation process, the first GPU is used to obtain a first reverse gradient corresponding to the first cross entropy and a second reverse gradient corresponding to the second cross entropy.

[0184] In a specific example of the disclosed solution, the second implicit state feature is located after the first implicit state feature in the sequence dimension of the target implicit state feature.

[0185] For the description of specific functions and examples of each unit of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0186] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0187] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0188] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0189] like Figure 13 As shown, device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. Various programs and data required for the operation of device 1300 can also be stored in RAM 1303. Computing unit 1301, ROM 1302, and RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to bus 1304.

[0190] Various components in device 1300 are connected to I / O interface 1305, including an input unit 1306, such as a keyboard and mouse; an output unit 1307, such as various types of displays and speakers; a storage unit 1308, such as a magnetic disk and optical disk; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1309 allows device 1300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0191] The computing unit 1301 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 1301 can be configured to perform the data processing method by any other appropriate means (e.g., by means of firmware).

[0192] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0194] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0196] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0197] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0198] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0199] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Obtaining, using a first computing unit of a first GPU, a first cross entropy corresponding to a first implicit state feature; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained by segmenting the target implicit state feature; and the target implicit state feature is obtained by feature processing text data or image data; Obtaining, using the first GPU, a second implicit state feature broadcasted by a second GPU among the plurality of GPUs, wherein the second implicit state feature is one of the plurality of sub-features; Obtaining, using a second computing unit of the first GPU, a second cross entropy corresponding to the second implicit state feature; A parallel processing result corresponding to a target implicit state feature is obtained using the first GPU and based on at least the first cross entropy and the second cross entropy.

2. The method according to claim 1, wherein The obtaining, by using the first computing unit of the first GPU, a first cross entropy corresponding to the first implicit state feature, includes: Processing the first implicit state feature using a first feature projection module in a first computing unit of the first GPU to obtain a first initial prediction value; The first initial prediction value is processed using a first loss calculation module in the first calculation unit to obtain a first cross entropy corresponding to the first implicit state feature.

3. The method according to claim 2, wherein: The first loss calculation module in the first calculation unit is used to process the first initial prediction value to obtain a first cross entropy corresponding to the first implicit state feature, including: Obtaining, using the first loss calculation module in the first computing unit, first feature data required for calculating the first cross entropy from at least the second GPU; A first cross entropy corresponding to the first implicit state feature is calculated using the first loss calculation module in the first computing unit and based on at least the first feature data obtained from the second GPU.

4. The method according to claim 1, wherein The obtaining, by using the first computing unit of the first GPU, a first cross entropy corresponding to the first implicit state feature, includes: Using a first computing unit of a first GPU, segment the first implicit state feature to obtain at least two first data blocks; Using a first computing unit of the first GPU, the at least two first data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each first data block; The sub-cross entropies corresponding to each first data block are fused using the first computing unit of the first GPU to obtain a first cross entropy corresponding to the first implicit state feature.

5. The method according to claim 1, wherein The obtaining, by using the second computing unit of the first GPU, a second cross entropy corresponding to the second implicit state feature, includes: Processing the second implicit state feature using a second feature projection module in a second computing unit of the first GPU to obtain a second initial prediction value; The second initial prediction value is processed using a second loss calculation module in the second calculation unit to obtain a second cross entropy corresponding to the second implicit state feature.

6. The method according to claim 5, wherein: The second loss calculation module in the second calculation unit is used to process the second initial prediction value to obtain a second cross entropy corresponding to the second implicit state feature, including: Obtaining, using a second loss calculation module in the second computing unit, second feature data required for calculating a second cross entropy from at least the second GPU; A second cross entropy corresponding to the second implicit state feature is calculated using a second loss calculation module in the second computing unit and based at least on the second feature data obtained from the second GPU.

7. The method according to claim 5 or 6, wherein: The obtaining, by using the second computing unit of the first GPU, a second cross entropy corresponding to the second implicit state feature, includes: Using a second computing unit of the first GPU, segment the second implicit state feature to obtain at least two second data blocks; Using a second computing unit of the first GPU, the at least two second data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each second data block; The sub-cross entropies corresponding to each second data block are fused using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

8. The method according to any one of claims 1 to 6, wherein: The method of obtaining a parallel processing result corresponding to a target implicit state feature by using the first GPU and based on at least the first cross entropy and the second cross entropy includes: In the forward propagation process, the total cross entropy corresponding to the target implicit state feature is obtained through the first GPU based on the first cross entropy and the second cross entropy.

9. The method according to any one of claims 1 to 6, wherein: The method of obtaining a parallel processing result corresponding to a target implicit state feature by using the first GPU and based on at least the first cross entropy and the second cross entropy includes: In the back propagation process, the first GPU is used to obtain a first reverse gradient corresponding to the first cross entropy and a second reverse gradient corresponding to the second cross entropy.

10. The method according to any one of claims 1 to 6, wherein: The second implicit state feature is located after the first implicit state feature in the sequence dimension of the target implicit state feature.

11. A data processing device comprising: a first processing unit, configured to obtain, using a first computing unit of a first GPU, a first cross entropy corresponding to a first implicit state feature; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained by segmenting the target implicit state feature; and the target implicit state feature is obtained by feature processing text data or image data; a communication unit, configured to obtain, using the first GPU, a second implicit state feature broadcasted by a second GPU among the plurality of GPUs; wherein the second implicit state feature is one of the plurality of sub-features; A second processing unit is configured to obtain a second cross entropy corresponding to a second implicit state feature using a second computing unit of the first GPU; The fusion processing unit is configured to obtain, using the first GPU and at least based on the first cross entropy and the second cross entropy, a parallel processing result corresponding to the target implicit state feature.

12. The device according to claim 11, wherein The first processing unit is specifically configured to: Processing the first implicit state feature using a first feature projection module in a first computing unit of the first GPU to obtain a first initial prediction value; The first initial prediction value is processed using a first loss calculation module in the first calculation unit to obtain a first cross entropy corresponding to the first implicit state feature.

13. The device according to claim 12, wherein The first processing unit is specifically configured to: Obtaining, using the first loss calculation module in the first computing unit, first feature data required for calculating the first cross entropy from at least the second GPU; A first cross entropy corresponding to the first implicit state feature is calculated using the first loss calculation module in the first computing unit and based on at least the first feature data obtained from the second GPU.

14. The device according to claim 11, wherein The first processing unit is specifically configured to: Using a first computing unit of a first GPU, segment the first implicit state feature to obtain at least two first data blocks; Using a first computing unit of the first GPU, the at least two first data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each first data block; The sub-cross entropies corresponding to each first data block are fused using the first computing unit of the first GPU to obtain a first cross entropy corresponding to the first implicit state feature.

15. The device according to claim 11, wherein The second processing unit is specifically configured to: Processing the second implicit state feature using a second feature projection module in a second computing unit of the first GPU to obtain a second initial prediction value; The second initial prediction value is processed using a second loss calculation module in the second calculation unit to obtain a second cross entropy corresponding to the second implicit state feature.

16. The device according to claim 15, wherein The second processing unit is specifically configured to: Obtaining, using a second loss calculation module in the second computing unit, second feature data required for calculating a second cross entropy from at least the second GPU; A second cross entropy corresponding to the second implicit state feature is calculated using a second loss calculation module in the second computing unit and based at least on the second feature data obtained from the second GPU.

17. The device according to claim 15 or 16, wherein The second processing unit is specifically configured to: Using a second computing unit of the first GPU, segment the second implicit state feature to obtain at least two second data blocks; Using a second computing unit of the first GPU, the at least two second data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each second data block; The sub-cross entropies corresponding to each second data block are fused using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

18. The device according to any one of claims 11 to 16, wherein: The fusion processing unit is specifically used to: In the forward propagation process, the total cross entropy corresponding to the target implicit state feature is obtained through the first GPU based on the first cross entropy and the second cross entropy.

19. The device according to any one of claims 11 to 16, wherein: The fusion processing unit is specifically used to: In the back propagation process, the first GPU is used to obtain a first reverse gradient corresponding to the first cross entropy and a second reverse gradient corresponding to the second cross entropy.

20. The device according to any one of claims 11 to 16, wherein: The second implicit state feature is located after the first implicit state feature in the sequence dimension of the target implicit state feature.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images