Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium, and relates to the technical field of data processing, in particular to the calculation fields of artificial intelligence, deep learning, large models and the like. According to the specific implementation scheme, a first calculation unit of a first GPU is utilized to obtain a first cross entropy corresponding to a first implicit state feature; the first GPU is one of a plurality of GPUs required for performing parallel processing on the target implicit state features; the first implicit state feature is one of a plurality of sub-features obtained by segmenting the target implicit state feature; using the first GPU to obtain a second implicit state feature broadcasted by the second GPU; the second implicit state feature is one of the plurality of sub-features; using a second calculation unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature; and obtaining a parallel processing result corresponding to the target implicit state feature by using the first GPU and at least based on the first cross entropy and the second cross entropy.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to technical fields such as artificial intelligence, deep learning, and large models. Background Art

[0002] Although the Large Language Model (LLM) has made significant breakthroughs in various fields, it will occupy too much video memory and even cause Out of Memory (OOM) problems when processing long sequences in distributed scenarios. Summary of the invention

[0003] The present disclosure provides a data processing method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided a data processing method, comprising:

[0005] Using a first computing unit of a first GPU, a first cross entropy corresponding to a first implicit state feature is obtained; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained after segmenting the target implicit state feature; the target implicit state feature is obtained after feature processing of text data or image data;

[0006] Using the first GPU, obtaining a second implicit state feature broadcasted by a second GPU among the multiple GPUs; wherein the second implicit state feature is one of the multiple sub-features;

[0007] Using a second computing unit of the first GPU, obtaining a second cross entropy corresponding to the second implicit state feature;

[0008] A parallel processing result corresponding to a target implicit state feature is obtained by using the first GPU and based at least on the first cross entropy and the second cross entropy.

[0009] According to another aspect of the present disclosure, there is provided a data processing device, comprising:

[0010] A first processing unit is used to obtain a first cross entropy corresponding to a first implicit state feature using a first computing unit of a first GPU; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained after segmenting the target implicit state feature; and the target implicit state feature is obtained after feature processing of text data or image data;

[0011] a communication unit, configured to obtain, by using the first GPU, a second implicit state feature broadcasted by a second GPU among the plurality of GPUs; wherein the second implicit state feature is one of the plurality of sub-features;

[0012] A second processing unit is used to obtain a second cross entropy corresponding to a second implicit state feature by using a second computing unit of the first GPU;

[0013] The fusion processing unit is used to obtain a parallel processing result corresponding to the target implicit state feature by using the first GPU and at least based on the first cross entropy and the second cross entropy.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] In this way, the disclosed scheme provides a refined scheme for the first GPU in a parallel scenario to obtain the parallel processing results corresponding to the target implicit state features (that is, the full features). In this scheme, there is no need to perform feature splicing between the first implicit state features and the second implicit state features processed in other GPUs. In other words, there is no need to perform the All Gather operation described above to obtain the parallel processing effect based on the full features (that is, the target implicit state features). In this way, the video memory resources required for parallel computing are reduced, thereby effectively avoiding the problem of insufficient memory when calculating the loss value of the target implicit state features due to the All Gather operation. On the basis of effectively ensuring distributed processing, it lays a foundation for further improving the processing efficiency of large models.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1(a) is a schematic diagram of a large model distributed training method for long text sequences;

[0024] FIG1( b ) is a schematic diagram of a parallel processing method based on a network structure composed of small operators;

[0025] Figure 2 is a schematic flow chart 1 of a data processing method according to an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of a parallel processing method based on a target fusion operator according to an embodiment of the present application;

[0027] FIG4( a ) is a schematic diagram of a scenario in a specific example of a data processing method according to an embodiment of the present application;

[0028] FIG4( b ) is a schematic diagram of a scenario in another specific example of a data processing method according to another embodiment of the present application;

[0029] Figure 5 It is a schematic diagram of a scenario of parallel processing in a back propagation process according to a data processing method of an embodiment of the present application;

[0030] Figure 6 This is a schematic flow chart of a data processing method according to an embodiment of the present application. Figure 2 ;

[0031] FIG. 7( a ) is a schematic diagram of the structure of a first computing unit in a target fusion operator according to an embodiment of the present application;

[0032] FIG7( b ) is a schematic diagram of the structure of a first computing unit in a target fusion operator according to another embodiment of the present application;

[0033] Figure 8 This is a schematic flow chart of a data processing method according to an embodiment of the present application. Figure 3 ;

[0034] FIG9( a ) is a schematic diagram of the structure of a second computing unit in a target fusion operator according to an embodiment of the present application;

[0035] FIG9( b ) is a schematic diagram of the structure of a second computing unit in a target fusion operator according to another embodiment of the present application;

[0036] FIG10( a ) is a schematic diagram of a parallel processing scenario in an example of a forward propagation process of a data processing method according to an embodiment of the present application;

[0037] FIG10( b ) is a schematic diagram of a parallel processing scenario in an example of a forward propagation process according to a data processing method according to another embodiment of the present application;

[0038] FIG11( a ) is a schematic diagram of a parallel processing scenario in an example of a back propagation process according to a data processing method according to an embodiment of the present application;

[0039] FIG11( b ) is a schematic diagram of a scenario of parallel processing in an example of a back-propagation process according to a data processing method according to another embodiment of the present application;

[0040] Fig.12 is a structural schematic diagram of a data processing device according to an embodiment of the present application;

[0041] Fig.13 It is a block diagram of an electronic device used to implement the data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0043] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0044] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0045] The following is a description of the related technologies of the embodiments of the present disclosure. The following related technologies are optional solutions that can be arbitrarily combined with the technical solutions of the embodiments of the present disclosure, and they all belong to the protection scope of the embodiments of the present disclosure.

[0046] In the large model scenario facing long text sequences, processing can be performed in a distributed manner. For example, in the distributed scenario of collective communication of large-scale clusters, as shown in Figure 1(a), in the application scenario of the transformer model using tensor parallelism (TP) and sequence parallelism (SP), for example, in the model training scenario, for an accelerator card (such as a GPU), an all gather operation is usually required to collect the hidden state features (such as the hidden state features with tensor dimensions of [B, S / tp, H]) learned on other accelerator cards (such as GPUs) to obtain a full amount of hidden state (such as the hidden state features with tensor dimensions of [B, S, H]). Here, B represents the batch size, that is, the number of sequences processed in a single training iteration, S represents the sequence dimension, that is, the length of the sequence, H represents the length of the hidden state features, and tp represents the tensor parallel dimension, that is, the number of hidden state features obtained after segmentation.

[0047] Furthermore, the parallel computing head (or parallel head) module is used to project the full amount of Hidden State into the output space, and the logical logarithm value (Logits) of the tensor dimension [B, S, V / tp] is calculated, and then the parallel cross entropy module is used to calculate the cross entropy loss value (loss) of the tensor dimension [B, S, 1] based on Logits, so as to facilitate the subsequent optimization of the model based on the loss. Here, V represents the vocabulary dimension, that is, the number of words or characters in the vocabulary.

[0048] It should be noted that as the sequence length increases, for example, to 128K or even larger, the full amount of Hidden State collected by AllGather will occupy a huge amount of video memory space, and even cause out of memory (OOM) problems when calculating logits and loss. For example, the vocabulary dimension (V) in the Large Language Model Meta Artificial Intelligence (LLAMA) 3.2 is 128K, the length of the implicit state feature (H) is 8192, and the tensor parallel dimension (tp) is 8. At this time, if the 16-bit Brain Floating Point 16 (BF16) data format is used, the sequence length (S) of the vocabulary is 1024K, the batch size (B) is 1, and the video memory size occupied by Logits is 32GB.

[0049] Therefore, to solve the above problem, in one example, the video memory problem caused by parallel computing can be optimized through a network structure composed of small operators. For example, as shown in Figure 1(b), due to the use of tensor parallelism and sequence parallelism, the Hidden State needs to be split on the sequence dimension S, and the tensor dimension of each sub-feature obtained after the split is [S / tp, H]; further, through the All Gather operation, the full amount of Hidden State (whose tensor dimension is [S, H]) is collected on each accelerator card (for example, GPU 0, GPU 1, etc.). Here, in one example, the Parallel Head module can use column slicing to split the weight (Weight, abbreviated as W) from the vocabulary dimension. For example, the Weight with a tensor of [H, V] is split into a Weight with a tensor dimension of [H, V / tp]. At this time, the Parallel Head module can be used on each accelerator card to perform matrix product processing on the Hidden State with a tensor dimension of [S, H] and the Weight with a tensor dimension of [H, V / tp] to obtain Logits [S, V / tp] in the split state of the vocabulary dimension, and then the Parallel Cross Entropy module is used to perform parallel loss calculation based on Logits [S, V / tp] and Label [S, V] to obtain loss [S, 1].

[0050] In the above scheme, although the network structure composed of small operators can alleviate the problem of insufficient video memory to a certain extent, there are still problems such as unstable video memory control and low computing efficiency, great optimization difficulty, and difficulty in ensuring the accuracy of the trained model.

[0051] Based on this, the disclosed solution provides a tensor parallel and sequence parallel processing solution. Compared with the existing solution, the disclosed solution can complete the loss calculation of long sequences without using the All Gather operation, which greatly improves the computational efficiency of parallel computing. At the same time, it also reduces the occupied video memory resources, thereby effectively solving the problem of insufficient memory when calculating losses.

[0052] Specifically, Figure 2 1 is a schematic flow chart of a data processing method according to an embodiment of the present application. The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters and other electronic devices.

[0053] Further, the method includes at least part of the following contents. Figure 2 As shown, including:

[0054] Step S201: using a first computing unit of a first GPU to obtain a first cross entropy corresponding to a first implicit state feature.

[0055] Here, the first GPU is one of the multiple GPUs required for parallel processing of the target implicit state feature; the first implicit state feature is one of the multiple sub-features obtained after splitting the target implicit state feature (also known as Hidden State) (for example, splitting according to the sequence dimension).

[0056] Furthermore, in one example, the target implicit state feature is obtained by performing feature processing on text data or image data.

[0057] For example, in one example, feature learning is performed on text data or image data to obtain a Hidden State with a tensor dimension of [S, H] (corresponding to the target implicit state feature above). At this time, the Hidden State with a tensor dimension of [S, H] is split on the sequence dimension to obtain multiple sub-features corresponding to the Hidden State. For example, the tensor dimension of each sub-feature can be [S / tp, H], where tp represents the tensor parallel dimension (that is, the number of implicit state features obtained after splitting).

[0058] Step S202: using the first GPU, obtaining a second implicit state feature broadcasted by a second GPU among the multiple GPUs.

[0059] Here, the second implicit state feature is one of the multiple sub-features.

[0060] Further, it can be understood that the second implicit state feature is different from the first implicit state feature. In this way, it is convenient to use multiple GPUs to process the target implicit state feature in parallel, thereby improving processing efficiency.

[0061] Step S203: using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

[0062] It should be pointed out that the "cross entropy" referred to in the present disclosure can be specifically understood as a cross entropy loss value (which can be abbreviated as loss), that is, a loss value calculated using a cross entropy loss function.

[0063] Step S204: using the first GPU and obtaining a parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy.

[0064] It should be noted that, in this example, since the parallel processing result is obtained by processing the first implicit state feature and the second implicit state feature using the first GPU, the parallel processing result can be understood as the parallel processing result of the target implicit state feature on the first GPU.

[0065] In this way, the disclosed scheme provides a refined scheme for the first GPU in a parallel scenario to obtain the parallel processing results corresponding to the target implicit state features (that is, the full features). In this scheme, there is no need to perform feature splicing between the first implicit state features and the second implicit state features processed in other GPUs. In other words, there is no need to perform the All Gather operation described above to obtain the parallel processing effect based on the full features (that is, the target implicit state features). In this way, the video memory resources required for parallel computing are reduced, thereby effectively avoiding the problem of insufficient memory when calculating the loss value of the target implicit state features due to the All Gather operation. On the basis of effectively ensuring distributed processing, it lays a foundation for further improving the processing efficiency of large models.

[0066] Furthermore, since the disclosed solution can utilize multiple computing units to perform parallel computing on the implicit state features obtained after segmentation, compared with the parallel computing solution of the prior art that collects a full amount of implicit state features through the All Gather operation, the disclosed solution simplifies the parallel computing process, reduces the overall complexity of parallel computing and the difficulty of optimization, and, while effectively ensuring the efficiency of parallel computing, effectively reduces the peak value of the video memory of the first GPU, providing strong support for improving the model accuracy of large models.

[0067] For example, GPU0 and GPU1 are used to process the target implicit state feature in parallel. At this time, the target implicit state feature can be divided into two parts in the sequence dimension, namely the first implicit state feature and the second implicit state feature. At this time, Figure 3As shown, the disclosed solution can directly input each implicit state feature (such as the first implicit state feature and the second implicit state feature) in the sequence dimension segmentation state into the computing unit (also referred to as the target fusion operator, for example, it can be specifically fused parallel cross entropy FusedHeadParallelCrossEntropy) of the corresponding GPU (such as GPU0 or GPU1). In this way, the parallel processing results of the target implicit state feature (that is, the full feature) on each GPU are obtained. Compared with the parallel processing solution shown in Figure 1(b), the parallel processing efficiency of the disclosed solution is higher, which reduces the overall complexity and optimization difficulty of parallel computing. At the same time, while effectively ensuring the parallel computing efficiency, it also effectively reduces the peak value of the video memory of each GPU.

[0068] Further, continue with Figure 3 Taking the example shown in FIG. 4( a) as an example, the first computing unit in GPU 0 (corresponding to the first GPU above) is used to obtain the first cross entropy corresponding to the sub-feature X0 (corresponding to the first implicit state feature above) of the target implicit state feature; and, GPU 0 is used to obtain the sub-feature X1 (corresponding to the second implicit state feature above) of the target implicit state feature sent by GPU 1 (corresponding to the second GPU above) in the form of broadcast communication, and the second computing unit in GPU 0 is used to obtain the second cross entropy corresponding to the sub-feature X1; further, the aggregation unit in GPU 0 is used to aggregate the obtained first cross entropy and the second cross entropy to obtain the parallel processing result corresponding to the target implicit state feature. In this way, the video memory resources required for parallel computing are effectively reduced, laying a foundation for the subsequent improvement of the processing efficiency of large models.

[0069] It should be noted that the number of GPUs required for parallel processing of the target implicit state feature is related to the parallel dimension of the tensor (i.e., tp). In other words, the number of GPUs required for parallel processing is related to the number of sub-features obtained after segmenting the target implicit state feature. For example, in one example, if the number of GPUs required for parallel computing is N, the target implicit state feature can be segmented and N sub-features can be obtained. At this time, the data processing method described above may further specifically include: using a first computing unit of a first GPU to obtain a first cross entropy corresponding to a first implicit state feature; using the first GPU to obtain a second implicit state feature broadcasted by each of N-1 second GPUs; using N-1 second computing units of the first GPU to perform parallel processing on N-1 second implicit state features to obtain a second cross entropy corresponding to each second implicit state feature; and then using the first GPU, and based on the first cross entropy and the second cross entropy corresponding to each second implicit state feature, obtaining a parallel processing result corresponding to the target implicit state feature. In this way, multiple computing units are used to process each sub-feature in parallel, thereby reducing the peak value of the video memory of the first GPU and improving the parallel processing efficiency.

[0070] For example, taking the case of using three GPUs to process the three sub-features obtained after segmentation in parallel, as shown in Figure 4(b), the computing unit 1 in GPU 0 is used to obtain the cross entropy 1 corresponding to the sub-feature X0; GPU 0 is used to obtain the sub-feature X1 sent by GPU 1 in the form of broadcast communication, and the computing unit 2 in GPU 0 is used to obtain the cross entropy 2 corresponding to the sub-feature X1; GPU 0 is used to obtain the sub-feature X2 sent by GPU 2 in the form of broadcast communication, and the computing unit 3 in GPU 0 is used to obtain the cross entropy 3 corresponding to the sub-feature X2; further, the aggregation unit in GPU 0 is used to aggregate the first cross entropy corresponding to the sub-feature X0, the second cross entropy corresponding to the sub-feature X1, and the second cross entropy corresponding to the sub-feature X2 to obtain the parallel processing result corresponding to the target implicit state feature. In this way, the peak value of the video memory of the first GPU in the parallel computing process is effectively reduced, laying a foundation for the subsequent improvement of the training efficiency of large models.

[0071] It should be noted that the above is only an example. In actual applications, the number of sub-features obtained after segmenting the target implicit state features and the number of GPUs may be one-to-one corresponding or not. They can be set according to actual conditions, and the present disclosure does not impose any specific restrictions on this.

[0072] Furthermore, in a specific example, the second implicit state feature is located after the first implicit state feature in the sequence dimension of the target implicit state feature, so that it is convenient to use multiple GPUs for parallel processing from the sequence dimension, thereby effectively realizing sequence parallelism.

[0073] Further, in a specific example, the above-described method of using the first GPU and obtaining the parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy (for example, step S204) may specifically include:

[0074] In the forward propagation process, the total cross entropy corresponding to the target implicit state feature is obtained through the first GPU based on the first cross entropy and the second cross entropy.

[0075] That is to say, in one example, in the forward propagation process, the parallel processing result obtained by using the above steps S201 to S204 can be specifically the total cross entropy corresponding to the target implicit state feature on the first GPU. The specific process details can be referred to the example of Figure 4(a), which will not be repeated here.

[0076] In this way, the disclosed solution quickly obtains the total cross entropy in the forward propagation through tensor parallel and sequence parallel processing solutions. The above process effectively reduces the GPU memory peak value during the parallel computing process, thus laying the foundation for the subsequent improvement of the processing efficiency of large models.

[0077] Alternatively, in another specific example, the above-described method of using the first GPU and obtaining the parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy (for example, step S204) may specifically include:

[0078] In the back propagation process, the first GPU is used to obtain a first reverse gradient corresponding to the first cross entropy and a second reverse gradient corresponding to the second cross entropy.

[0079] That is, in one example, in the back propagation process, the parallel processing results obtained by using the above steps S201 to S204 can be specifically the first reverse gradient corresponding to the first cross entropy and the second reverse gradient of the second cross entropy. Figure 3 Take the example shown as an example, Figure 5As shown, in the back propagation process, the recompute technology is used, and the first cross entropy is obtained through the first computing unit in GPU0. Here, the step of obtaining the first cross entropy in the back propagation process is similar to the step of obtaining the first cross entropy in the forward propagation process, and will not be repeated here. Further, after the first cross entropy is obtained through the first computing unit in GPU0, the first gradient calculation unit is used, and the first reverse gradient corresponding to the first cross entropy is obtained according to the relevant data (such as the first relevant data) required to calculate the first reverse gradient, that is, the first reverse gradient for the sub-feature X0 is obtained, for example, including the reverse gradient X0 and the reverse gradient W0.

[0080] Similarly, the recompute technology is used, and the second cross entropy is obtained through the second computing unit in GPU0. Here, the step of obtaining the second cross entropy in the back propagation process is similar to the step of obtaining the second cross entropy in the forward propagation process, and will not be repeated here. Further, after the second cross entropy is obtained through the second computing unit in GPU0, the second gradient calculation unit is used, and the second reverse gradient corresponding to the second cross entropy is obtained according to the relevant data (such as the second relevant data) required for calculating the second reverse gradient, that is, the second reverse gradient for the sub-feature X1 is obtained, for example, including the reverse gradient X1 and the reverse gradient W0.

[0081] In this way, the disclosed solution can recompute the technology to quickly obtain the reverse gradient in the back propagation. Moreover, the above process effectively reduces the peak value of the GPU memory during the parallel computing process, thus laying the foundation for the subsequent improvement of the processing efficiency of large models.

[0082] Figure 6 This is a schematic flow chart of a data processing method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figures 2 to 5 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0083] Further, the method includes at least part of the following contents. Figure 6 As shown, including:

[0084] Step S601: using a first feature projection module in a first computing unit of a first GPU to process a first implicit state feature to obtain a first initial prediction value.

[0085] Here, the first GPU is one of the multiple GPUs required for parallel processing of the target implicit state feature; the first implicit state feature is one of the multiple sub-features obtained after splitting the target implicit state feature (also known as Hidden State) (for example, splitting according to the sequence dimension).

[0086] Furthermore, in one example, the target implicit state feature is obtained by performing feature processing on text data or image data.

[0087] It should be noted that the relevant contents about the number of GPUs and sub-features can be referred to the above examples and will not be elaborated here.

[0088] Furthermore, the first initial prediction value may specifically be the first logical logarithm value output by the last layer in the model, for example, specifically a logits value that has not been normalized.

[0089] Step S602: Using the first loss calculation module in the first calculation unit, the first initial prediction value is processed to obtain a first cross entropy corresponding to the first implicit state feature.

[0090] That is to say, in the disclosed solution, first, the first implicit state feature is processed using the first feature projection module in the first computing unit in the first GPU (for example, the first implicit state feature is multiplied by the weight in the first GPU) to obtain a first initial prediction value, and then the first loss calculation module in the first computing unit is used to obtain the first cross entropy based on the obtained first initial prediction value. In this way, it is convenient to obtain the parallel processing result corresponding to the target implicit state feature based on the first cross entropy and the second cross entropy.

[0091] For example, continuing with Figure 3 Taking the example shown in FIG. 7( a) as an example, the first computing unit in GPU 0 (corresponding to the first GPU above) is used to process the sub-feature X0 of the target implicit state feature, which may specifically include: firstly using the first feature projection module in the first computing unit to process the sub-feature X0 to obtain the first Logits (corresponding to the first initial prediction value above), and then using the first loss calculation module in the first computing unit (such as the cross entropy loss calculation module), and according to the obtained first Logits, calculating the first cross entropy corresponding to the sub-feature X0. In this way, data support is provided for the subsequent parallel processing results corresponding to the target implicit state feature on the first GPU.

[0092] Step S603: using the first GPU, obtaining a second implicit state feature broadcasted by a second GPU among the multiple GPUs.

[0093] Here, the second implicit state feature is one of the multiple sub-features.

[0094] It should be noted that the relevant content about the second implicit state feature can be referred to the above example and will not be elaborated here.

[0095] Step S604: using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

[0096] Step S605: using the first GPU and obtaining a parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy.

[0097] In this way, the disclosed solution provides a refined solution for using the first computing unit to process the first implicit state feature, that is, using multiple modules in the first computing unit to process the first implicit state feature, thereby effectively reducing the peak value of the video memory of the first GPU during the calculation process, and further improving the efficiency of parallel computing, thereby laying the foundation for the subsequent training and optimization of large models.

[0098] Further, in a specific example, the first cross entropy may be obtained in the following manner; specifically, the above-described processing of the first initial prediction value using the first loss calculation module in the first calculation unit to obtain the first cross entropy corresponding to the first implicit state feature (for example, step S602) may specifically include:

[0099] Step S602-1: using the first loss calculation module in the first calculation unit, obtaining first feature data required for calculating the first cross entropy from at least the second GPU.

[0100] Step S602-2: Utilizing the first loss calculation module in the first calculation unit and based at least on the first feature data obtained from the second GPU, calculate a first cross entropy corresponding to the first implicit state feature.

[0101] That is to say, in the process of processing the first initial prediction value using the first loss calculation module in the first computing unit, it is also necessary to use the first loss calculation module to obtain the first feature data required for calculating the first cross entropy from the second GPU, and then the first loss calculation module obtains the first cross entropy based on the acquired first feature data and the first initial prediction value.

[0102] It should be pointed out that when the number of sub-features obtained by segmenting the target implicit state feature is N (N is greater than 2), the first feature data required for calculating the first cross entropy may be N-1. At this time, the first loss calculation module in the first calculation unit can be used to obtain the first feature data required for calculating the first cross entropy from each second GPU of the N-1 second GPUs, and then the first loss calculation module in the first calculation unit is used again, and based on the first feature data obtained from each second GPU, the first cross entropy corresponding to the first implicit state feature is calculated. In this way, the accuracy and reliability of the subsequent parallel processing results are ensured.

[0103] In this way, the disclosed solution further provides a refined solution for obtaining the first cross entropy. The solution is simple and efficient, and can quickly calculate the first cross entropy. Moreover, the feature data used in the above process does not need to be spliced ​​with the features of other second GPUs. Therefore, compared with the existing solution, the peak value of the video memory of the first GPU in the calculation process is effectively reduced, thereby effectively ensuring efficient parallel computing and laying a foundation for the subsequent training and optimization of large models.

[0104] In a specific example, in order to further reduce the peak value of the video memory during the calculation process, the first implicit state may be divided into blocks, the divided data blocks may be processed in parallel, and the processing results corresponding to each data block may be fused to obtain the first cross entropy; specifically, the above-mentioned first calculation unit of the first GPU is used to obtain the first cross entropy corresponding to the first implicit state feature (for example, step S201), which may specifically include:

[0105] Step a: using the first computing unit of the first GPU to segment the first implicit state feature to obtain at least two first data blocks.

[0106] Step b: using the first computing unit of the first GPU to perform parallel processing on the at least two first data blocks to obtain a sub-cross entropy corresponding to each first data block.

[0107] Step c: using the first computing unit of the first GPU to fuse the sub-cross entropies corresponding to each first data block to obtain a first cross entropy corresponding to the first implicit state feature.

[0108] In this way, the disclosed scheme first divides the first implicit state feature, and then processes the divided data blocks in parallel to obtain sub-cross entropies of each data block in parallel, and then fuses the sub-cross entropies of each data block to obtain the first cross entropy. In this way, the disclosed scheme uses subbatch technology to further reduce the peak value of the video memory of the first GPU in the intermediate calculation process, thereby further saving the required resources and time. At the same time, it also further improves the parallel computing efficiency, laying the foundation for subsequent obtaining the parallel processing results corresponding to the target implicit state features and improving the training efficiency of the model.

[0109] It should be noted that the above-mentioned process of "processing the divided data blocks in parallel" can also be applied to the detailed process of step S601 to step S602. For example, after the first implicit state feature is divided to obtain M (an integer greater than or equal to 2) first data blocks, the M first feature projection sub-modules in the first calculation unit are used to process the M first data blocks in parallel to obtain the first initial prediction sub-value of each first data block, and then the M first loss calculation sub-modules in the first calculation unit are used to process the first initial prediction sub-value of each first data block in parallel to obtain the sub-cross entropy of each first data block; finally, the aggregation module in the first calculation unit is used to aggregate the sub-cross entropies of each first data block to obtain the first cross entropy corresponding to the first implicit state feature.

[0110] For example, continuing with Figure 3 Taking the example shown in FIG. 7( b ), the first computing unit in GPU 0 (corresponding to the first GPU above) is used to split the sub-feature X0 into data blocks X00 and X01, and then the first feature projection submodule 1 in the first computing unit is used to obtain the first Logits subvalue 1 of the data block X00, and then the first loss calculation submodule 1 in the first computing unit is used to process the first Logits subvalue 1 to obtain the sub-cross entropy (i.e., sub-cross entropy 01) of the data block X00; accordingly, the first feature projection submodule 2 in the first computing unit is used to obtain the first Logits subvalue 2 of the data block X01, and then the first loss calculation submodule 2 in the first computing unit is used to process the first Logits subvalue 2 to obtain the sub-cross entropy (i.e., sub-cross entropy 02) of the data block X01, and then the aggregation module in the first computing unit is used to aggregate the sub-cross entropies of each data block to obtain the first cross entropy corresponding to the first implicit state feature. In this way, the peak value of the video memory of the first GPU during the calculation process is further reduced.

[0111] Figure 8 This is a schematic flow chart of a data processing method according to an embodiment of the present application. Figure 3The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 2 The related contents of the method shown in FIG. 7 can also be applied to this example, and the related contents will not be described in detail in this example.

[0112] Further, the method includes at least part of the following contents. Figure 8 As shown, including:

[0113] Step S801: using a first computing unit of a first GPU to obtain a first cross entropy corresponding to a first implicit state feature.

[0114] Here, the first GPU is one of the multiple GPUs required for parallel processing of the target implicit state feature; the first implicit state feature is one of the multiple sub-features obtained after splitting the target implicit state feature (also known as Hidden State) (for example, splitting according to the sequence dimension).

[0115] Furthermore, in one example, the target implicit state feature is obtained by performing feature processing on text data or image data.

[0116] It should be noted that the relevant contents about the number of GPUs and sub-features can be referred to the above examples and will not be elaborated here.

[0117] Furthermore, the first initial prediction value may specifically be the first logical logarithm value output by the last layer in the model, for example, specifically a logits value that has not been normalized.

[0118] Step S802: using the first GPU, obtaining a second implicit state feature broadcasted by a second GPU among the multiple GPUs.

[0119] Here, the second implicit state feature is one of the multiple sub-features.

[0120] It should be noted that the relevant content about the second implicit state feature can be referred to the above example and will not be elaborated here.

[0121] Step S803: using the second feature projection module in the second computing unit of the first GPU to process the second implicit state feature to obtain a second initial prediction value.

[0122] Furthermore, the second initial prediction value may specifically be a second logical logarithm value output by the last layer in the model, for example, a logits value that has not been normalized.

[0123] Step S804: using the second loss calculation module in the second calculation unit, the second initial prediction value is processed to obtain a second cross entropy corresponding to the second implicit state feature.

[0124] That is to say, in the disclosed solution, first, the second implicit state feature is processed using the second feature projection module in the second computing unit in the first GPU (for example, the second implicit state feature is multiplied by the weight in the first GPU) to obtain a second initial prediction value, and then the second loss calculation module in the second computing unit is used to obtain the second cross entropy based on the obtained second initial prediction value. In this way, it is convenient to obtain the parallel processing result corresponding to the target implicit state feature based on the first cross entropy and the second cross entropy.

[0125] For example, continuing with Figure 3 Taking the example shown in FIG. 9(a) as an example, the second computing unit in GPU 0 (corresponding to the first GPU above) is used to process the sub-feature X1 of the target implicit state feature, which may specifically include: firstly using the second feature projection module in the second computing unit to process the sub-feature X1 to obtain the second Logits (corresponding to the second initial prediction value above), and then using the second loss calculation module in the second computing unit (such as the cross entropy loss calculation module), and according to the obtained second Logits, calculate the second cross entropy corresponding to the sub-feature X1. In this way, data support is provided for the subsequent parallel processing results corresponding to the target implicit state feature on the first GPU.

[0126] Step S805: using the first GPU and obtaining a parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy.

[0127] In this way, the disclosed solution provides a refined solution for using a second computing unit to process the second implicit state features, that is, using multiple modules in the second computing unit to process the second implicit state features. In this way, the peak value of the video memory of the first GPU during the calculation process is effectively reduced, and the efficiency of parallel computing is further improved, thereby laying the foundation for the subsequent training and optimization of large models.

[0128] Further, in a specific example, the second cross entropy can be obtained in the following manner; specifically, the second initial prediction value is processed by the second loss calculation module in the second calculation unit to obtain the second cross entropy corresponding to the second implicit state feature (for example, step S804), which can specifically include:

[0129] Step S804-1: using the second loss calculation module in the second computing unit, obtaining second feature data required for calculating the second cross entropy from at least the second GPU.

[0130] Step S804-2: Utilizing the second loss calculation module in the second computing unit and based at least on the second feature data obtained from the second GPU, calculate a second cross entropy corresponding to the second implicit state feature.

[0131] That is to say, in the process of processing the second initial prediction value using the second loss calculation module in the second computing unit, it is also necessary to use the second loss calculation module to obtain the second feature data required for calculating the second cross entropy from the second GPU, and then the second loss calculation module obtains the second cross entropy based on the acquired second feature data and the second initial prediction value.

[0132] It should be pointed out that when the number of sub-features obtained by segmenting the target implicit state feature is N (N is greater than 2), the second feature data required for calculating the second cross entropy may be N-1. At this time, the second loss calculation module in the second computing unit can be used to obtain the second feature data required for calculating the second cross entropy from each second GPU of the N-1 second GPUs, and then the second loss calculation module in the second computing unit is used again, and based on the second feature data obtained from each second GPU, the second cross entropy corresponding to the second implicit state feature is calculated. In this way, the accuracy and reliability of the subsequent parallel processing results are ensured.

[0133] In this way, the disclosed solution further provides a refined solution for obtaining the second cross entropy. The solution is simple and efficient, and can quickly calculate the second cross entropy. Moreover, the feature data used in the above process does not need to be spliced ​​with the features of other second GPUs. Therefore, compared with the existing solution, the peak value of the video memory of the first GPU in the calculation process is effectively reduced, thereby effectively ensuring efficient parallel computing and laying a foundation for the subsequent training and optimization of large models.

[0134] In a specific example, in order to further reduce the peak value of the video memory during the calculation process, the second implicit state can also be divided into blocks, the divided data blocks are processed in parallel, and the processing results corresponding to each data block are merged to obtain the second cross entropy; specifically, the above-mentioned use of the second computing unit of the first GPU to obtain the second cross entropy corresponding to the second implicit state feature (for example, step S203) can specifically include:

[0135] Step a: using the second computing unit of the first GPU to segment the second implicit state feature to obtain at least two second data blocks.

[0136] Step b: using the second computing unit of the first GPU to perform parallel processing on the at least two second data blocks to obtain a sub-cross entropy corresponding to each second data block.

[0137] Step c: using the second computing unit of the first GPU, the sub-cross entropies corresponding to each second data block are fused to obtain a second cross entropy corresponding to the second implicit state feature.

[0138] In this way, the disclosed scheme first divides the second implicit state feature, and then processes the divided data blocks in parallel to obtain sub-cross entropies of each data block in parallel, and then fuses the sub-cross entropies of each data block to obtain the second cross entropy. In this way, the disclosed scheme uses subbatch technology to further reduce the peak value of the video memory of the first GPU during the intermediate calculation process, thereby further saving the required resources and time. At the same time, it also further improves the parallel computing efficiency, laying the foundation for subsequent obtaining the parallel processing results corresponding to the target implicit state features and improving the training efficiency of the model.

[0139] It should be noted that the above-mentioned process of "processing the divided data blocks in parallel" can also be applied to the detailed process of step S803 to step S804. For example, after the second implicit state feature is divided to obtain M (an integer greater than or equal to 2) second data blocks, the M second feature projection sub-modules in the second computing unit are used to process the M second data blocks in parallel to obtain the second initial prediction sub-value of each second data block, and then the M second loss calculation sub-modules in the second computing unit are used to process the second initial prediction sub-value of each second data block in parallel to obtain the sub-cross entropy of each second data block; finally, the aggregation module in the second computing unit is used to aggregate the sub-cross entropy of each second data block to obtain the second cross entropy corresponding to the second implicit state feature.

[0140] For example, continuing with Figure 3Taking the example shown in FIG. 9( b ), the second computing unit in GPU 0 (corresponding to the first GPU above) is used to split the sub-feature X1 into data blocks X10 and X11, and the second feature projection submodule 1 in the second computing unit is used to obtain the second Logits sub-value 1 of the data block X10, and then the second loss calculation submodule 1 in the second computing unit is used to process the second Logits sub-value 1 to obtain the sub-cross entropy (i.e., sub-cross entropy 11) of the data block X10; accordingly, the second feature projection submodule 2 in the second computing unit is used to obtain the second Logits sub-value 2 of the data block X11, and then the second loss calculation submodule 2 in the second computing unit is used to process the second Logits sub-value 2 to obtain the sub-cross entropy (i.e., sub-cross entropy 12) of the data block X11, and then the aggregation module in the second computing unit is used to aggregate the sub-cross entropies of each data block to obtain the second cross entropy corresponding to the second implicit state feature. In this way, the peak value of the video memory of the first GPU during the calculation process is further reduced.

[0141] It should be noted that the disclosed solution mainly describes the processing flow of the first GPU in detail. In actual scenarios, the processing process of other GPUs (for example, the second GPU) is similar to that of the first GPU, and will not be described in detail here.

[0142] The following is a further detailed description of the disclosed solution in conjunction with FIG. 10(a), FIG. 10(b) and FIG. 11(a), FIG. 11(b); Figure 3Taking the example shown as an example, GPU0 and GPU1 are used to process the target implicit state features in parallel. At this time, the target implicit state features can be divided into two parts in the sequence dimension, namely the first implicit state features and the second implicit state features. Further, as shown in Figure 10(a), in the forward propagation process, the feature projection module 1 in GPU0 is used to perform matrix product processing on the sub-feature X0 and the weight 0 (also denoted as W0) in GPU 0 to obtain the initial prediction value 1, and then the loss calculation module 1 in GPU0 is used to calculate the cross entropy 1 according to the initial prediction value 1 and the feature data 1 obtained from GPU1. And using the feature projection module 2 in GPU0, the sub-feature X1 broadcasted by GPU1 is matrix-producted with the weight 0 in GPU 0 to obtain the initial prediction value 2, and then using the loss calculation module 2 in GPU0, the cross entropy 2 is calculated based on the initial prediction value 2 and the feature data 2 obtained from GPU1; further, in GPU 0, using the aggregation unit 1, the cross entropy 1 and the cross entropy 2 are aggregated to obtain the total cross entropy loss of the target implicit state feature on GPU0 (i.e., loss 1); similarly, the total cross entropy loss of the target implicit state feature on GPU1 (i.e., loss 2) can be obtained. Here, it can be understood that the processing flow in GPU1 is similar to that in GPU0, and will not be repeated here.

[0143] Furthermore, the disclosed solution can also divide the sub-features to obtain multiple data blocks, and then process each data block in parallel to obtain the cross entropy 1. Here, the calculation unit 1 is taken as an example for explanation. As shown in FIG10( b), the sub-feature X0 is divided into data block X00 and data block X01. At this time, the matrix product submodule 1 in GPU0 is used to perform matrix product processing on the data block X00 and the weight 0 to obtain the initial prediction value 01. Then, the loss calculation submodule 1 in GPU0 is used to obtain the sub-cross entropy according to the initial prediction value 01 and the loss obtained from GPU1. 01, the sub-cross entropy 01 corresponding to the data block X00 is obtained; similarly, the matrix product submodule 2 in GPU0 is used to perform matrix product processing on the data block X01 and the weight 0 to obtain the initial prediction value 02, and then the loss calculation submodule 2 in GPU0 is used to obtain the sub-cross entropy 02 corresponding to the data block X01 according to the initial prediction value 02 and the partial feature data 1 obtained from GPU1 for obtaining the sub-cross entropy 02; further, the sub-cross entropies corresponding to each data block are aggregated to obtain the cross entropy 1 corresponding to the sub-feature X0. Here, a similar method can be used to obtain cross entropy 2, cross entropy 3 and cross entropy 4, and then loss1 and loss2 are obtained, which will not be repeated here.

[0144] In summary, for the forward propagation process, the disclosed solution has the following advantages:

[0145] First, the processing flow is simplified. Specifically, the disclosed solution removes the All Gather operator, and can directly input the sub-features of the target implicit state feature in the sequence dimension segmentation state (for example, HiddenState[S / tp, H]), and then calculate the Logits and cross entropy loss values ​​in sequence according to the order of each sub-feature on the GPU to obtain the total cross entropy loss of the target implicit state feature on each GPU. In this way, the calculation process is simplified, the GPU memory peak value during the calculation process is effectively reduced, and the efficiency of parallel computing is improved.

[0146] Second, lower video memory. Specifically, the disclosed solution can also use Subbatch technology on the GPU to calculate the cross entropy of sub-features, that is, to divide the sub-features of the target implicit state feature again, such as dividing the sub-feature HiddenState[S / tp, H] into data blocks HiddenState[chunk_size, H], and then cyclically calculate the sub-cross entropy of each data block, and finally aggregate the sub-cross entropy of each data block in the sequence dimension to obtain the cross entropy of the sub-feature, thus further reducing the GPU video memory peak during the calculation process.

[0147] For another example, as shown in FIG11(a), in the back propagation process, the recompute technique is used, and the cross entropy 1 is obtained through the computing unit 1 in GPU0. Here, the step of obtaining the cross entropy 1 in the back propagation process is similar to the step of obtaining the cross entropy 1 in the forward propagation process shown in FIG10(a), and will not be repeated here. Further, after the cross entropy 1 is obtained through the computing unit 1 in GPU0, the gradient calculation unit 1 is used, and the reverse gradient X0-0 and the reverse gradient W0-0 for the sub-feature X0 on GPU0 are obtained according to the relevant data (for example, the first relevant data) required for calculating the subsequent reverse gradient. Similarly, the reverse gradient X1-0 and the reverse gradient W0-1 for the sub-feature X1 can also be obtained on GPU0, and the reverse gradient X0-1 and the reverse gradient W1-0 for the sub-feature X0, and the reverse gradient X1-1 and the reverse gradient W1-1 for the sub-feature X1 are obtained on GPU1.

[0148] Furthermore, the disclosed solution can also divide the sub-features to obtain multiple data blocks, and then process each data block in parallel to obtain the reverse gradient for the sub-features. Here, the calculation unit 1 is taken as an example for explanation, as shown in FIG11(b). Here, the specific process of FIG11(b) is similar to the processing flow of FIG10(b), and will not be repeated here.

[0149] In summary, for the back-propagation process, the disclosed solution adopts the Recompute technology, re-executes the forward propagation process, and then calculates the reverse gradient. Compared with caching the processing results of the forward propagation, the disclosed solution further reduces the GPU memory peak value during the calculation process, and at the same time, reduces the occupancy of the cache space.

[0150] It should be noted that the disclosed solution does not limit the specific calculation process of obtaining the above-mentioned Logits, cross entropy, and reverse gradient.

[0151] Furthermore, in summary, the disclosed solution has the following advantages:

[0152] First, the disclosed solution adopts segmentation and broadcast communication operations in the fusion operator (i.e., FusedHeadParallelCrossEntropy) to broadcast the data information in the current GPU to other GPUs, so that other GPUs can use the acquired data information to calculate matrix products and cross entropy loss values. Therefore, the disclosed solution can significantly reduce the peak value of video memory.

[0153] Second, the disclosed solution uses Subbatch technology within the fusion operator to further reduce the peak value of video memory in the intermediate calculation process.

[0154] Third, when performing back propagation, the disclosed solution also uses Recompute technology, combined with operator fusion technology and Subbatch technology, to collaboratively achieve the reduction of the peak memory usage during training of long sequence models, thereby enabling the disclosed solution to support distributed training of models with more than one trillion parameters in scenarios where the sequence length is greater than 128K.

[0155] The disclosed solution also provides a data processing device, such as Fig.12 As shown, including:

[0156] The first processing unit 1201 is used to obtain a first cross entropy corresponding to a first implicit state feature using a first computing unit of a first GPU; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained after segmenting the target implicit state feature; the target implicit state feature is obtained after feature processing of text data or image data;

[0157] A communication unit 1202 is configured to obtain, by using the first GPU, a second implicit state feature broadcasted by a second GPU among the multiple GPUs; wherein the second implicit state feature is one of the multiple sub-features;

[0158] The second processing unit 1203 is used to obtain a second cross entropy corresponding to the second implicit state feature by using the second computing unit of the first GPU;

[0159] The fusion processing unit 1204 is used to obtain a parallel processing result corresponding to the target implicit state feature by using the first GPU and at least based on the first cross entropy and the second cross entropy.

[0160] In a specific example of the disclosed solution, the first processing unit is specifically used to:

[0161] Using a first feature projection module in a first computing unit of a first GPU, the first implicit state feature is processed to obtain a first initial prediction value;

[0162] The first initial prediction value is processed using a first loss calculation module in the first calculation unit to obtain a first cross entropy corresponding to the first implicit state feature.

[0163] In a specific example of the disclosed solution, the first processing unit is specifically used to:

[0164] Using the first loss calculation module in the first calculation unit, obtaining at least first feature data required for calculating the first cross entropy from the second GPU;

[0165] A first cross entropy corresponding to the first implicit state feature is calculated using the first loss calculation module in the first calculation unit and based at least on the first feature data obtained from the second GPU.

[0166] In a specific example of the disclosed solution, the first processing unit is specifically used to:

[0167] Using a first computing unit of a first GPU, segmenting the first implicit state feature to obtain at least two first data blocks;

[0168] Using a first computing unit of the first GPU, the at least two first data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each first data block;

[0169] The sub-cross entropies corresponding to each first data block are fused using the first computing unit of the first GPU to obtain a first cross entropy corresponding to the first implicit state feature.

[0170] In a specific example of the disclosed solution, the second processing unit is specifically used to:

[0171] Processing the second implicit state feature using a second feature projection module in a second computing unit of the first GPU to obtain a second initial prediction value;

[0172] The second initial prediction value is processed using a second loss calculation module in the second calculation unit to obtain a second cross entropy corresponding to the second implicit state feature.

[0173] In a specific example of the disclosed solution, the second processing unit is specifically used to:

[0174] Using a second loss calculation module in the second calculation unit, obtaining second feature data required for calculating a second cross entropy from at least the second GPU;

[0175] A second cross entropy corresponding to the second implicit state feature is calculated using a second loss calculation module in the second computing unit and based at least on the second feature data obtained from the second GPU.

[0176] In a specific example of the disclosed solution, the second processing unit is specifically used to:

[0177] Using a second computing unit of the first GPU, segmenting the second implicit state feature to obtain at least two second data blocks;

[0178] Using a second computing unit of the first GPU, the at least two second data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each second data block;

[0179] The sub-cross entropies corresponding to each second data block are fused using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

[0180] In a specific example of the disclosed solution, the fusion processing unit is specifically used to:

[0181] In the forward propagation process, the total cross entropy corresponding to the target implicit state feature is obtained through the first GPU based on the first cross entropy and the second cross entropy.

[0182] In a specific example of the disclosed solution, the fusion processing unit is specifically used to:

[0183] In the back propagation process, the first GPU is used to obtain a first reverse gradient corresponding to the first cross entropy and a second reverse gradient corresponding to the second cross entropy.

[0184] In a specific example of the disclosed solution, the second implicit state feature is located after the first implicit state feature in the sequence dimension of the target implicit state feature.

[0185] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0186] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0187] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0188] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0189] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0190] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0191] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via ROM 1302 and / or communication unit 1309. When the computer program is loaded into RAM 1303 and executed by the computing unit 1301, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0192] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0194] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0196] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0197] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0198] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0199] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Using a first computing unit of a first GPU, a first cross entropy corresponding to a first implicit state feature is obtained; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained after segmenting the target implicit state feature; the target implicit state feature is obtained after feature processing of text data or image data; Using the first GPU, obtaining a second implicit state feature broadcasted by a second GPU among the multiple GPUs; wherein the second implicit state feature is one of the multiple sub-features; Using a second computing unit of the first GPU, obtaining a second cross entropy corresponding to the second implicit state feature; A parallel processing result corresponding to a target implicit state feature is obtained by using the first GPU and based at least on the first cross entropy and the second cross entropy.

2. The method according to claim 1, wherein: The method of using the first computing unit of the first GPU to obtain a first cross entropy corresponding to the first implicit state feature includes: Using a first feature projection module in a first computing unit of a first GPU, the first implicit state feature is processed to obtain a first initial prediction value; The first initial prediction value is processed using a first loss calculation module in the first calculation unit to obtain a first cross entropy corresponding to the first implicit state feature.

3. The method according to claim 2, wherein: The first loss calculation module in the first calculation unit is used to process the first initial prediction value to obtain a first cross entropy corresponding to the first implicit state feature, including: Using the first loss calculation module in the first calculation unit, obtaining at least first feature data required for calculating the first cross entropy from the second GPU; A first cross entropy corresponding to the first implicit state feature is calculated using the first loss calculation module in the first calculation unit and based at least on the first feature data obtained from the second GPU.

4. The method according to any one of claims 1 to 3, wherein: The method of using the first computing unit of the first GPU to obtain a first cross entropy corresponding to the first implicit state feature includes: Using a first computing unit of a first GPU, segmenting the first implicit state feature to obtain at least two first data blocks; Using a first computing unit of the first GPU, the at least two first data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each first data block; The sub-cross entropies corresponding to each first data block are fused using the first computing unit of the first GPU to obtain a first cross entropy corresponding to the first implicit state feature.

5. The method according to claim 1, wherein: The method of using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature includes: Processing the second implicit state feature using a second feature projection module in a second computing unit of the first GPU to obtain a second initial prediction value; The second initial prediction value is processed using a second loss calculation module in the second calculation unit to obtain a second cross entropy corresponding to the second implicit state feature.

6. The method according to claim 5, wherein: The second loss calculation module in the second calculation unit is used to process the second initial prediction value to obtain a second cross entropy corresponding to the second implicit state feature, including: Using a second loss calculation module in the second calculation unit, obtaining second feature data required for calculating a second cross entropy from at least the second GPU; A second cross entropy corresponding to the second implicit state feature is calculated using a second loss calculation module in the second computing unit and based at least on the second feature data obtained from the second GPU.

7. The method according to claim 5 or 6, wherein: The method of using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature includes: Using a second computing unit of the first GPU, segmenting the second implicit state feature to obtain at least two second data blocks; Using a second computing unit of the first GPU, the at least two second data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each second data block; The sub-cross entropies corresponding to each second data block are fused using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

8. The method according to any one of claims 1 to 7, wherein: The method of using the first GPU and obtaining a parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy includes: In the forward propagation process, the total cross entropy corresponding to the target implicit state feature is obtained through the first GPU based on the first cross entropy and the second cross entropy.

9. The method according to any one of claims 1 to 7, wherein: The method of using the first GPU and obtaining a parallel processing result corresponding to the target implicit state feature based at least on the first cross entropy and the second cross entropy includes: In the back propagation process, the first GPU is used to obtain a first reverse gradient corresponding to the first cross entropy and a second reverse gradient corresponding to the second cross entropy.

10. The method according to claim 1, wherein: The second implicit state feature is located after the first implicit state feature in the sequence dimension of the target implicit state feature.

11. A data processing device, comprising: A first processing unit is used to obtain a first cross entropy corresponding to a first implicit state feature using a first computing unit of a first GPU; wherein the first GPU is one of a plurality of GPUs required for parallel processing of a target implicit state feature; the first implicit state feature is one of a plurality of sub-features obtained after segmenting the target implicit state feature; and the target implicit state feature is obtained after feature processing of text data or image data; a communication unit, configured to obtain, by using the first GPU, a second implicit state feature broadcasted by a second GPU among the plurality of GPUs; wherein the second implicit state feature is one of the plurality of sub-features; A second processing unit is used to obtain a second cross entropy corresponding to a second implicit state feature by using a second computing unit of the first GPU; The fusion processing unit is used to obtain a parallel processing result corresponding to the target implicit state feature by using the first GPU and at least based on the first cross entropy and the second cross entropy.

12. The device according to claim 11, wherein The first processing unit is specifically configured to: Using a first feature projection module in a first computing unit of a first GPU, the first implicit state feature is processed to obtain a first initial prediction value; The first initial prediction value is processed using a first loss calculation module in the first calculation unit to obtain a first cross entropy corresponding to the first implicit state feature.

13. The device according to claim 12, wherein: The first processing unit is specifically configured to: Using the first loss calculation module in the first calculation unit, obtaining at least first feature data required for calculating the first cross entropy from the second GPU; A first cross entropy corresponding to the first implicit state feature is calculated using the first loss calculation module in the first calculation unit and based at least on the first feature data obtained from the second GPU.

14. The device according to any one of claims 11 to 13, wherein: The first processing unit is specifically configured to: Using a first computing unit of a first GPU, segmenting the first implicit state feature to obtain at least two first data blocks; Using a first computing unit of the first GPU, the at least two first data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each first data block; The sub-cross entropies corresponding to each first data block are fused using the first computing unit of the first GPU to obtain a first cross entropy corresponding to the first implicit state feature.

15. The device according to claim 11, wherein The second processing unit is specifically configured to: Processing the second implicit state feature using a second feature projection module in a second computing unit of the first GPU to obtain a second initial prediction value; The second initial prediction value is processed using a second loss calculation module in the second calculation unit to obtain a second cross entropy corresponding to the second implicit state feature.

16. The device according to claim 15, wherein: The second processing unit is specifically configured to: Using a second loss calculation module in the second calculation unit, obtaining second feature data required for calculating a second cross entropy from at least the second GPU; A second cross entropy corresponding to the second implicit state feature is calculated using a second loss calculation module in the second computing unit and based at least on the second feature data obtained from the second GPU.

17. The device according to claim 15 or 16, wherein: The second processing unit is specifically configured to: Using a second computing unit of the first GPU, segmenting the second implicit state feature to obtain at least two second data blocks; Using a second computing unit of the first GPU, the at least two second data blocks are processed in parallel to obtain a sub-cross entropy corresponding to each second data block; The sub-cross entropies corresponding to each second data block are fused using the second computing unit of the first GPU to obtain a second cross entropy corresponding to the second implicit state feature.

18. The device according to any one of claims 11 to 17, wherein: The fusion processing unit is specifically used for: In the forward propagation process, the total cross entropy corresponding to the target implicit state feature is obtained through the first GPU based on the first cross entropy and the second cross entropy.

19. The device according to any one of claims 11 to 17, wherein: The fusion processing unit is specifically used for: In the back propagation process, the first GPU is used to obtain a first reverse gradient corresponding to the first cross entropy and a second reverse gradient corresponding to the second cross entropy.

20. The device according to claim 11, wherein The second implicit state feature is located after the first implicit state feature in the sequence dimension of the target implicit state feature.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images