Data processing method and related apparatus

Abstract

Provided in the embodiments of the present application are a data processing method and a related apparatus. The method comprises: sharding in a first dimension first data to obtain M pieces of first sub-data; sending M first sub-sequences to a second device; receiving M second sub-sequences sent by the second device; and, on the basis of the M second sub-sequences, obtaining second data. The embodiments of the present application can improve the training efficiency and inference efficiency of mixture of experts.

Description

Data processing method and related apparatus

[0001] The present application claims priority to the Chinese patent application No. 202411366633.4, filed on September 27, 2024, and entitled "A data processing method and related apparatus", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular, to a data processing method and related apparatus. BACKGROUND

[0003] The hybrid expert model can be regarded as a sparse model obtained by replacing part of the feedforward neural network (FNN) layers in the traditional transformer model with expert layers. Each expert layer in the hybrid expert model contains multiple experts, and each expert can be regarded as an independent neural network. The hybrid expert model can select a suitable expert in the expert layer to process the input data of the expert layer according to the characteristics of the input data, that is, part of the neural network parameters can be dynamically activated instead of all the parameters.

[0004] Since the more experts in the expert layer, the more the overall parameters of the model, the training and inference of the current hybrid expert model usually adopts the expert parallel mode, that is, the multiple experts in each expert layer in the hybrid expert model are respectively deployed in multiple devices, and other functional layers are replicated on each device. Communication connections are established between the multiple devices, and data transmission is performed through the communication network between the devices, so as to share multiple experts of one expert layer, thereby reducing the storage pressure of a single device. In the expert parallel scenario, the time of cross-device communication will affect the training efficiency and inference efficiency of the hybrid expert model. SUMMARY

[0005] The embodiments of the present application provide a data processing method and related apparatus, which can be applied to the field of artificial intelligence to save the communication time between devices, thereby improving the training efficiency and inference efficiency of the hybrid expert model.

[0006] In a first aspect, the present application provides a data processing method, which is applied to a first device including a first expert, and the method comprises:

[0007] The first data is split in a first dimension to obtain M first sub-data, wherein the first data includes multiple sequences, each sequence in the multiple sequences corresponds to an expert, and each sequence is used to represent the quantization information of a token in the first data, and the first dimension is the direction of each sequence;

[0008] sending, to the second device, M first sub-sequences, wherein each of the M first sub-data comprises a first sub-sequence, each of the M first sub-sequences comprises partial data of the first sequence, the first sequence is one of the plurality of sequences in the first data, and the first sequence corresponds to the second expert of the second device;

[0009] receiving M second sub-sequences sent by the second device, wherein the M second sub-sequences are sequences obtained by processing the M first sub-sequences by the second expert respectively;

[0010] obtaining second data according to the M second sub-sequences, wherein the second data is data obtained by processing the first data by the first expert and / or the second expert, and a second sequence of the second data in the first dimension is a sequence obtained by processing the first sequence by the second expert.

[0011] In the above method, the first device performs splitting processing on the first data in the first dimension during cross-device communication, so that the first device can send M first sub-sequences of the first sequence to the second device (for example, the M first sub-sequences can be sent synchronously, or another first sub-sequence can be sent in the sending time period of one first sub-sequence), which is beneficial to obtaining the processing result (i.e., the second data) corresponding to the first data subsequently.

[0012] As can be seen, compared with the method in which the first device directly sends the first sequence to the second device, the above method can reduce the amount of data transmitted during cross-device communication, avoid data redundancy, save the time of cross-device communication, and thus improve the training efficiency and inference efficiency of the hybrid expert model.

[0013] In an optional implementation, the lengths of the M first sub-sequences are equal, and the sending, to the second device, of the M first sub-sequences comprises:

[0014] sending, by the kth first computing unit of the first device, the kth first sub-sequence to the kth second computing unit of the second device, so as to realize the sending, by the M first computing units of the first device, of the M first sub-sequences to the M second computing units of the second device, wherein the kth first sub-sequence comprises data belonging to the first sequence in the kth first sub-data of the M first sub-data;

[0015] The receiving, from the second device, of the M second sub-sequences comprises:

[0016] receiving, by the kth first computing unit, the kth second sub-sequence sent by the kth second computing unit, so as to realize the receiving, by the M first computing units, of the M second sub-sequences sent by the M second computing units, wherein the kth second sub-sequence is a sequence obtained by processing the kth first sub-sequence by the second expert, and k is an integer from 1 to M in turn.

[0017] In the method, the M first computing units of the first device and the M computing units of the second device are one-to-one corresponding, and each first computing unit only transmits data to the second computing unit corresponding to the first computing unit during cross-device communication. In addition, the first device splits the first data, so that the M first computing units synchronously transmit M first sub-sequences obtained by splitting the first sequence to the corresponding second computing units respectively, which is beneficial to obtaining the processing result (i.e., the second data) corresponding to the first data.

[0018] Therefore, compared with the method in which each first computing unit transmits the first sequence to the corresponding second computing unit, the method can reduce the amount of data transmitted during cross-device communication, avoid data redundancy, save the time of cross-device communication, and thus improve the training efficiency and inference efficiency of the hybrid expert model.

[0019] In an optional implementation, the second sequence is a sequence obtained by splicing the M second sub-sequences in the first dimension.

[0020] In another optional implementation, the second data is obtained according to the M second sub-sequences, including:

[0021] receiving, by the kth first computing unit, M-1 second sub-sequences transmitted by M-1 first computing units other than the kth first computing unit from among the M first computing units;

[0022] splicing, by the kth first computing unit, the received M second sub-sequences in the first dimension to obtain a second sequence;

[0023] obtaining the second data according to the M second sub-sequences.

[0024] In the method, the M first computing units of the first device transmit data to each other during intra-device communication, and after the M computing units of the first device receive the processing results (i.e., the M second sub-sequences) corresponding to the M first sub-sequences respectively transmitted by the corresponding second computing units, the M second sub-sequences are spliced by each first computing unit to obtain the processing result (i.e., the second sequence) corresponding to the first sequence through data transmission between intra-device computing units. Therefore, the method can restore the split sub-sequences to complete sequences through intra-device communication, thereby enhancing the accuracy of the training result and inference result of the hybrid expert model.

[0025] In another optional implementation, before the first data is split in the first dimension, the method further includes:

[0026] determining that the expert matching the first sequence is the second expert.

[0027] In another optional implementation, the method further includes:

[0028] determining a first expert matching a third sequence of the first data in the first dimension, wherein the third sequence is one of the plurality of sequences in the first data, the third sequence corresponds to the first expert, the M first sub-data respectively comprise M third sub-sequences, each of the M third sub-sequences comprises partial data of the third sequence, and the third sequence is used to represent quantization information of one token in the first input data;

[0029] processing, by the kth first computing unit, the kth third sub-sequence to obtain a kth fourth sub-sequence, wherein the kth third sub-sequence comprises data belonging to the third sequence in the kth first sub-data of the M first sub-data, and the kth fourth sub-sequence is a sequence obtained by processing the kth third sub-sequence by the first expert;

[0030] obtaining the second data according to the M second sub-sequences, comprising:

[0031] obtaining the second data according to the M second sub-sequences and the M fourth sub-sequences, wherein a fourth sequence of the second data in the first dimension is a sequence obtained by processing the third sequence by the first expert.

[0032] In yet another optional implementation, the fourth sequence is a sequence obtained by splicing the M fourth sub-sequences in the first dimension.

[0033] In yet another optional implementation, processing, by the kth first computing unit, the kth third sub-sequence to obtain a kth fourth sub-sequence, comprising:

[0034] receiving, by the kth first computing unit, M-1 third sub-sequences sent by M-1 first computing units, wherein the M-1 first computing units are computing units other than the kth first computing unit in the M first computing units;

[0035] processing, by the kth first computing unit, the third sequence to obtain a kth fifth sequence, wherein the third sequence is a sequence obtained by splicing the kth third sub-sequence and M-1 third sub-sequences in the first dimension;

[0036] receiving, by the kth first computing unit, M-1 fifth sequences sent by M-1 first computing units;

[0037] obtaining, by the kth first computing unit, the kth fourth sub-sequence according to the fourth sequence, wherein the fourth sequence is a sequence obtained by summing the M fifth sequences.

[0038] In the method, after the first device determines that the expert matching the third sequence in the first data is the first expert in the first device, the first device performs splitting processing on the first data, allocates M third sub-sequences obtained by splitting the third sequence to the M first computing units of the first device respectively, and the M first computing units perform splicing on the M third sub-sequences to obtain the third sequence, facilitating subsequent processing. Therefore, the method can restore the split sub-sequences to complete sequences through intra-device communication, thereby enhancing the accuracy of the training result and the inference result of the mixed expert model.

[0039] In yet another optional implementation, the first data includes a plurality of sequences in the first dimension, and the third sequence is one of the plurality of sequences.

[0040] In yet another optional implementation, the method further includes:

[0041] The kth first computing unit receives the kth sixth sub-sequence sent by the kth second computing unit, so as to realize that the M first computing units receive M sixth sub-sequences sent by the M second computing units, wherein the M sixth sub-sequences have equal lengths, the kth sixth sub-sequence includes data belonging to the sixth sequence in the kth second sub-data of the M second sub-data, the M second sub-data is obtained by the second device splitting the third data in the first dimension, the third data includes a plurality of sequences, each sequence of the plurality of sequences corresponds to an expert, each sequence in the third sequence is used to represent quantization information of a token in the third data, the sixth sequence is one of the plurality of sequences in the third data, and the sixth sequence corresponds to the first expert.

[0042] The kth first computing unit processes the kth sixth sub-sequence to obtain a kth seventh sub-sequence, wherein the kth sixth sub-sequence includes data belonging to the sixth sequence in the kth second sub-data of the M second sub-data, and the kth seventh sub-sequence is a sequence obtained by processing the kth sixth sub-sequence by the first expert.

[0043] The kth first computing unit sends the kth seventh sub-sequence to the kth second computing unit.

[0044] In yet another optional implementation, a communication mechanism between the kth first computing unit in the first device and the kth second computing unit in the second device is all-to-all communication, and a communication mechanism between the M first computing units in the first device includes an all-gather communication mechanism and an all-reduce communication.

[0045] In yet another optional implementation, the communication network of the all-to-all communication mechanism is a remote direct memory access (RDMA) network, and the communication network of the all-reduce communication mechanism is a cache coherence (HCCS) network.

[0046] In yet another optional implementation, the first input data includes a tensor representing association information between tokens in each of a plurality of token sets, dimensions of the first input data include a first dimension, a second dimension, and a third dimension, a length of the first input data on the first dimension is equal to a number of values used to quantify a token in the plurality of tokens, a length of the first input data on the second dimension is equal to a number of the plurality of tokens, and a third length of the first input data on the third dimension is equal to a number of the plurality of token sets.

[0047] In yet another optional implementation, the first data is the first input data.

[0048] In yet another optional implementation, the first data is any one of N first data obtained by splitting the first input data, and N is a positive integer greater than or equal to 2.

[0049] In yet another optional implementation, the first device sequentially starts processing the N first data to obtain the second data according to a first time offset, and the first time offset is a time for a single first data to perform a kth first sub-sequence operation from the kth first computing unit of the first device to the kth second computing unit of the second device.

[0050] The embodiments of the present application can realize concurrent execution of the communication process and the calculation process in the data processing process, thereby masking the communication time, shortening the end-to-end execution time, and further improving the training efficiency and inference efficiency of the hybrid expert model.

[0051] In yet another optional implementation, a length of the first input data on the first dimension is equal to a length of the first data on the first dimension, a length of the first input data on the second dimension is W1 times a length of the first data on the second dimension, a length of the first input data on the third dimension is W2 times a length of the first data on the third dimension, N is equal to a product of W1 and W2, W1 is a positive integer, and W2 is a positive integer greater than or equal to 2.

[0052] In yet another optional implementation, the hybrid expert model includes at least one expert layer and at least one feed-forward neural network (FFN) layer, the at least one expert layer includes a first expert layer, and each expert layer in the at least one expert layer includes at least one expert.

[0053] In yet another possible implementation manner, the kth first computing unit is one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU).

[0054] In a second aspect, the embodiments of the present application provide a data processing apparatus, which can be the first device in the first aspect or any possible implementation manner of the first aspect, and wherein the data processing apparatus comprises a module for performing the method in the first aspect or any possible implementation manner of the first aspect.

[0055] In a third aspect, the embodiments of the present application provide a data processing apparatus, which comprises a transceiver, a processor and a storage, the transceiver is configured to perform receiving and / or transmitting operations under the control of the processor, the storage is configured to store a computer program, and the processor is configured to invoke the computer program, and wherein:

[0056] The processor is configured to split the first data in a first dimension to obtain M first sub-data, wherein the first data comprises a plurality of sequences, each sequence in the plurality of sequences corresponds to an expert, each sequence is used to represent quantization information of a token in the first data, and the first dimension is a direction of each sequence.

[0057] The transceiver is configured to send M first sub-sequences to the second device, wherein each first sub-data in the M first sub-data comprises a first sub-sequence, each first sub-sequence in the M first sub-sequences comprises partial data of a first sequence, the first sequence is one sequence in the plurality of sequences in the first data, and the first sequence corresponds to a second expert of the second device.

[0058] The transceiver is configured to receive M second sub-sequences sent by the second device, wherein the M second sub-sequences are sequences obtained by processing the M first sub-sequences by the second expert respectively.

[0059] The processor is configured to obtain second data according to the M second sub-sequences, wherein the second data is data obtained by processing the first data by the first expert and / or the second expert, and a second sequence of the second data in the first dimension is a sequence obtained by processing the first sequence by the second expert.

[0060] In a fourth aspect, the embodiments of the present application provide a data processing apparatus, which comprises a logic circuit and an interface, and the logic circuit and the interface are coupled; the interface is configured to input and / or output information, and wherein:

[0061] The logic circuit is configured to perform the method in the first aspect or any possible implementation manner of the first aspect.

[0062] In a fifth aspect, an embodiment of the present application provides a data processing system, characterized by comprising the first device in the first aspect or any possible implementation manner of the first aspect, and the second device in the first aspect or any possible implementation manner of the first aspect.

[0063] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium for storing a computer program, wherein the computer program is executed to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0064] The method, system and device provided by any possible implementation manner of the second aspect to the sixth aspect of the present application have the beneficial effects of the technical solutions provided by the first aspect and any possible implementation manner of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0065] The following describes the drawings used in the embodiments of the present application.

[0066] FIG. 1a is a structural schematic diagram of a transformer model provided by an embodiment of the present application;

[0067] FIG. 1b is a structural schematic diagram of a hybrid expert model provided by an embodiment of the present application;

[0068] FIG. 2a is an architectural schematic diagram of a data processing system provided by an embodiment of the present application;

[0069] FIG. 2b is an architectural schematic diagram of another data processing system provided by an embodiment of the present application;

[0070] FIG. 3 is a flow schematic diagram of a data processing method provided by an embodiment of the present application;

[0071] FIG. 4 is a structural schematic diagram of data provided by an embodiment of the present application;

[0072] FIG. 5a is a process schematic diagram of data processing provided by an embodiment of the present application;

[0073] FIG. 5b is another process schematic diagram of data processing provided by an embodiment of the present application;

[0074] FIG. 5c is another process schematic diagram of data processing provided by an embodiment of the present application;

[0075] FIG. 6 is a process schematic diagram of data processing related to the related art;

[0076] FIG. 7 is another process schematic diagram of a data processing method provided by an embodiment of the present application;

[0077] FIG. 8 is a flow diagram of another data processing method according to an embodiment of the present application;

[0078] FIG. 9a is a process diagram of another data processing method according to an embodiment of the present application;

[0079] FIG. 9b is a process diagram of another data processing method according to an embodiment of the present application;

[0080] FIG. 10 is a structural diagram of a data processing apparatus according to an embodiment of the present application;

[0081] FIG. 11 is a structural diagram of another data processing apparatus according to an embodiment of the present application;

[0082] FIG. 12 is a structural diagram of another data processing apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0083] The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0084] The transformer model is a deep learning model for processing sequential data, which can be widely applied in image processing, natural language processing, sound recognition, etc. The hybrid expert model can be regarded as a sparse model obtained by replacing part of the feedforward neural network (FNN) layers in the transformer model with expert layers. For ease of understanding, an exemplary hybrid expert model is explained below in conjunction with FIG. 1a and FIG. 1b.

[0085] Referring to FIG. 1a, which is a structural diagram of a transformer model according to an embodiment of the present application. As shown in FIG. 1a, the transformer model 10 includes multiple encoding layers and multiple decoding layers, and input data is processed by the multiple encoding layers and the multiple decoding layers in sequence to obtain output data. FIG. 1a illustrates that the transformer model 10 includes 6 encoding layers and 6 decoding layers. The 6 encoding layers have the same structure but different parameters, and each encoding layer includes a self-attention layer and an FNN layer. For example, the encoding layer 101 includes a first self-attention layer 102 and a first FNN layer 103. The 6 decoding layers have the same structure but different parameters, and each decoding layer includes a self-attention layer, a decoding attention layer, and an FNN layer. For example, the decoding layer 104 includes a second self-attention layer 105, a decoding attention layer 106, and a second FNN layer 107.

[0086] The mixed expert model can be obtained by replacing part of the FFN layers in the transformer model with expert layers. Each expert layer contains multiple experts. For example, referring to FIG. 1b, which is a structural schematic diagram of a mixed expert model according to an embodiment of the present application. Compared with the transformer model 10 shown in FIG. 1a, the mixed expert model 11 shown in FIG. 1b can be regarded as a model obtained by replacing the first FFN layer 103 in the transformer model 10 with a first expert layer 110. The first expert layer 110 in the encoding layer 101 includes V experts, i.e., expert E1, expert E2, …, expert EV. V V is a positive integer greater than 1. Each expert in an expert layer can be regarded as an independent neural network. For example, an expert can be an FFN, a more complex network structure, or even a mixed expert model itself, thereby forming a hierarchical model structure.

[0087] The first expert layer 110 can select a suitable expert in the expert layer to process the input data of the expert layer according to the characteristics of the input data. Optionally, the first expert layer 110 can include a gating network 111 (or routing), which can assign multiple tokens to corresponding experts for processing according to the data characteristics of each token in the input data of the expert layer, and perform residual connection and layer normalization (indicated by a symbol in the figure) on the results obtained by processing and the input data of the expert layer, to obtain the output data of the expert layer. In order to facilitate the processing steps of residual connection and layer normalization in the expert layer, the dimensions of the input data and the output data of the expert layer of the mixed expert model 11 are equal.

[0088] It should be noted that in the field of artificial intelligence, a token represents a smallest data unit in a data processing process. For example, for text data, a text token usually represents a word, a punctuation mark, a letter, or a number in the text. The proportion of the expert layer and the FFN layer in the mixed expert model is not strictly limited according to the embodiments of the present application. FIG. 1b is only an example, and in actual applications, the proportion of the expert layer and the FFN layer can be adjusted according to actual needs. As described above, the transformer model is used to process sequence data, and therefore the mixed expert model can also be used to process sequence data. The type of task processed by the mixed expert model is not strictly limited according to the embodiments of the present application.

[0089] Therefore, the hybrid expert model can dynamically activate part of the neural network parameters instead of all the parameters, so as to effectively train or infer with much less computing resources than a dense model (such as a transformer model) requires. That is, under the same computing budget, the hybrid expert model can significantly expand the size of the model or the data set, especially in the pre-training stage, and the hybrid expert model can usually reach the same training level faster than the dense model.

[0090] However, the more experts in the expert layer, the more parameters the model has as a whole. Currently, the training and / or inference of the hybrid expert model usually adopts an expert-parallel manner, that is, the multiple experts in each expert layer of the hybrid expert model are respectively deployed in multiple devices, and the other functional layers are replicated on each device to reduce the storage pressure of a single device.

[0091] Hereinafter, the expert-parallel manner will be further explained by taking an example of the multiple experts in the hybrid expert model 11 shown in FIG. 1b being respectively deployed in multiple devices. Please refer to FIG. 2a, which is an architecture schematic diagram of a data processing system according to an embodiment of the present application.

[0092] As shown in FIG. 2a, the data processing system 20 includes X devices, which include a device 201, a device 202, …, and a device 20X, X being a positive integer greater than 1. The multiple experts in each expert layer of the hybrid expert model 11 can be respectively deployed in the X devices. Optionally, the number of experts on each device is equal. Optionally, each of the X devices can be a device with strong computing capability, for example, a server or a server cluster composed of multiple servers.

[0093] FIG. 2a illustrates the deployment of the V experts in the first expert layer 110 of the hybrid expert model 11 when V is equal to X. Specifically, the input data of the hybrid expert model 11 is evenly split into X sub-input data, which are respectively taken as the input data of the X devices. Each of the X devices includes the first self-attention layer 102 and the gating network 111 (or routing) in the first expert layer 110 of the hybrid expert model 11. Moreover, each device includes one expert in the first expert layer 110, for example, the device 201 includes the expert E1, the device 202 includes the expert E2, …, and the device 20X includes the expert EX. V It can be understood that the input data of the first expert layer 110 in FIG. 1b can be regarded as being evenly split into X sub-input data in FIG. 2a, which are respectively processed by the X devices, and the output data of the first expert layer 110 in FIG. 1b can be regarded as being evenly split into X sub-output data in FIG. 2a, which are respectively output by the X devices.

[0094] The communication connection is established between any two of the X devices, and data transmission is performed through the inter-device communication network, so that the multiple tokens in the input data of one expert layer can be distributed to the corresponding experts for processing, thereby sharing the multiple experts of one expert layer. For example, if the device 201 determines that the expert corresponding to a token in one sub-input data of the first expert layer is the expert E2 in the device 202, the device 201 sends the token to the device 202 through the inter-device communication network, and the device 202 can process the token based on the expert E2 and send the processing result of the token to the device 201 through the inter-device communication network. If the device 201 determines that the expert corresponding to a token in one sub-input data of the first expert layer is the expert E1 in the device 201, the device 201 can process the token based on the expert E1 to obtain the processing result of the token.

[0095] It should be noted that any two of the X devices can perform data transmission through the inter-device communication network, thereby sharing the multiple experts of one expert layer. In FIG. 2a, only the data transmission between the device 201 and the device 202 is shown. The deployment of the other expert layers in the hybrid expert model 11 except the first expert layer 110 in the X devices can refer to the deployment of the first expert layer 110 shown in FIG. 2a. One token in the input data of one expert layer can be distributed to multiple experts. The specific rules of distribution are not limited in the embodiments of the present application. In actual application, one token is usually distributed to one expert or two experts for processing. The number of experts in each device of the X devices is not strictly limited in the embodiments of the present application. FIG. 2a is only an example.

[0096] In an optional implementation, each of the X devices includes a computing unit, and the multiple computing units are used to perform the computation of the experts belonging to one expert layer in the device.

[0097] In an optional implementation, each of the X devices includes multiple computing units, and the multiple computing units are used to jointly perform the computation of the experts belonging to one expert layer in the device. The communication connection can be established between any two of the multiple computing units to perform data transmission. The number of computing units included in one device is not limited in the embodiments of the present application. For ease of understanding, for example, refer to FIG. 2b, which is an architecture schematic diagram of another data processing system provided by the embodiments of the present application.

[0098] As can be seen from FIG. 2a and FIG. 2b, each of the X devices includes 4 computing units, and a communication connection is established between any two of the 4 computing units (FIG. 2b schematically shows the communication connection between multiple computing units in a single device by bidirectional solid arrows). Each of the 4 computing units is configured to perform partial computation of an expert belonging to the first expert layer 110 in the device where the computing unit is located, to obtain a partial computation result corresponding to the expert, and to obtain other partial computation results corresponding to the expert by communicating with other 3 computing units of the 4 computing units, so as to obtain a final computation result corresponding to the expert.

[0099] For example, the device 201 of the X devices includes 4 computing units, i.e., a computing unit 2011, a computing unit 2012, a computing unit 2013, and a computing unit 2014, and the 4 computing units are respectively configured to perform partial computation of an expert E1 belonging to the first expert layer 110 in the device 201. Optionally, the expert E1 includes 4 sub-experts, i.e., a sub-expert E 11 , a sub-expert E 12 , a sub-expert E 13 , and a sub-expert E 14 . The 4 sub-experts are distributed and deployed on the 4 computing units, so that each computing unit is configured to perform computation of one sub-expert, for example, the computing unit 2011 is configured to perform computation of the sub-expert E 11 .

[0100] As described above, the number of experts of the single expert layer included in each of the X devices is not strictly limited. When the number of experts of the single expert layer included in each of the X devices is greater than 1, one computing unit in the device can perform partial computation of each expert of the single expert layer. For example, the computing unit 2011 can not only be configured to perform computation of the sub-expert E 11 of the first expert E1, but also be configured to perform computation of one sub-expert of another expert.

[0101] In addition, a communication connection can also be established between any two of the X devices, so as to share multiple experts in the first expert layer 110 (FIG. 2b schematically shows the communication connection between multiple computing units in any two devices by bidirectional dashed arrows).

[0102] Optionally, the computing unit in the embodiments of the present application can be a physical computing resource, for example, can be a graphics processing unit (GPU), a data processing unit (DPU), a digital signal processing (DSP), a neural-network processing unit (NPU), an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA), etc. Optionally, the computing unit in the embodiments of the present application can also be a virtual computing resource, for example, can be a virtual CPU (also referred to as vCPU), a virtual GPU (also referred to as vGPU), a virtual DSP (also referred to as vDSP), a virtual NPU (also referred to as vNPU), a virtual ASIC (also referred to as vASIC), or a virtual FPGA (also referred to as vFPGA), etc.

[0103] It should be noted that for one expert layer in the hybrid expert model 11 except the first expert layer 110, each device of the X devices can also include a plurality of computing units in addition to the computing unit shown in FIG. 2b, for performing the calculation of the experts belonging to the expert layer in the device where the computing unit is located. That is to say, for each expert layer in the hybrid expert model 11, each device of the X devices can be provided with a plurality of corresponding computing units to perform the corresponding computing tasks.

[0104] It can be known from FIGS. 2a and 2b that for the data processing process of a single expert layer in the hybrid expert model, the plurality of devices can adopt the hierarchical communication strategy shown in FIG. 2b, that is, the data transmission between any two devices of the plurality of devices is performed through the inter-device communication network, the data transmission between the plurality of computing units in each device is performed through the intra-device communication network, and the plurality of computing units in each device jointly perform the calculation of each expert in the single expert layer. Then, the data processing time of the single expert layer in the hybrid expert model is related to the time of communication between the plurality of devices (i.e., the time of cross-device communication), the time of communication between the plurality of computing units in each device (i.e., the time of intra-device communication), and the time of calculation of each computing unit in each device (i.e., the time of intra-device calculation).

[0105] Compared with the efficiency of intra-device communication and the efficiency of intra-device calculation, the rate of cross-device communication is low. Therefore, in the scenario of expert parallelism, the time of cross-device communication has a great influence on the data processing time of a single expert layer in the hybrid expert model, and further influences the training efficiency and inference efficiency of the hybrid expert model.

[0106] In view of this, the embodiments of the present application provide a data processing method and related device, which can save the time of cross-device communication, thereby improving the training efficiency and inference efficiency of the hybrid expert model.

[0107] Please refer to FIG. 3, which is a flowchart of a data processing method provided by an embodiment of the present application. The method can be implemented based on the architecture shown in FIG. 2a and FIG. 2b, or can be implemented based on other architectures. The method includes but is not limited to the following steps:

[0108] Step S301: The first device splits the first input data of the first expert layer in the first dimension to obtain M first sub-data.

[0109] The first device includes a first expert, which is one expert of the first expert layer in the hybrid expert model, and the first expert layer also includes a second expert in the second device. The hybrid expert model includes at least one expert layer. The first expert layer is any one of the at least one expert layer, and the first expert layer includes at least two experts. For the explanation of the hybrid expert model, please refer to the explanation of the hybrid expert model in the foregoing, which will not be repeated here.

[0110] It can be understood that the first device and the second device can be any two devices of the X devices in the embodiments shown in FIG. 2a and FIG. 2b, the hybrid expert model can be the hybrid expert model 11 in the embodiment shown in FIG. 2a, and the first expert layer can be the first expert layer 110 in the embodiment shown in FIG. 2a. The number of experts in the first device and the second device is not strictly limited in the embodiments of the present application. The first expert is any one expert in the first device, and the second expert is any one expert in the second device. For ease of description, the method embodiment shown in FIG. 3 will be explained below by taking the first expert layer in the hybrid expert model as an example, which has only two experts (i.e. the first expert and the second expert), and the two experts are deployed on the first device and the second device respectively.

[0111] It can be understood in combination with FIG. 2a that the first input data of the first expert layer is the output data obtained by processing the input data of the first device by the functional layer (such as the self-attention layer) arranged in front of the first expert layer in the first device in turn, and the input data of the first device can be regarded as a sub-input data of the input data of the hybrid expert model.

[0112] Specifically, the first input data is a tensor used to represent association information between multiple tokens in each token set. The dimensions of the first input data include a first dimension, a second dimension, and a third dimension. It can be understood that the length of the first input data in the first dimension is equal to the number of all values used to quantify a token, the length of the first input data in the second dimension is equal to the number of tokens in a token set, and the length of the first input data in the third dimension is equal to the number of token sets.

[0113] The data structure of the first input data is further explained by taking the scenario of processing a language translation task by using a hybrid expert model as an example. In the above scenario, the first input data can be a tensor used to represent a paragraph of text.

[0114] Optionally, the paragraph of text can include B sentences, and each sentence includes C characters, that is, every C characters in the paragraph of text form a group to be a sentence. Optionally, the sentence in the embodiment of the present application can be regarded as a sentence divided from the paragraph of text according to the number of characters. It can be understood that a sentence can be regarded as a token set, and a character can be regarded as a token. That is to say, the first input data is a tensor used to represent association information between multiple characters in each sentence. Optionally, the association information can include context association information, nonlinear association information, and the like.

[0115] For example, the first input data can be represented by the tensor shown in FIG. 4. In FIG. 4, a small cube is used to represent a value in the first input data H0. The dimensions of the first input data H0 include a first dimension, a second dimension, and a third dimension. The first input data H0 includes 6*6 sequences in the first dimension, and each sequence is used to represent quantification information of a character in a sentence. For example, the tensor H0 includes a sequence h0 (that is, a sequence composed of the values represented by the gray cubes in FIG. 4) in the multiple sequences in the first dimension, and the sequence h0 can be used to represent quantification information of a “write” character.

[0116] As shown in FIG. 4, the length L1 of the first input data H0 in the first dimension can be equal to the number of all values used to quantify a character, the length L2 of the first input data H0 in the second dimension is equal to the number C of characters included in a sentence, and the length L3 of the first input data H0 in the third dimension is equal to the number B of sentences in a paragraph of text. The length L1 of the first input data H0 in the first dimension is related to an algorithm for processing language coding of a character, and can be set according to actual requirements.

[0117] Optionally, the first dimension can also be referred to as a hidden dimension, the second dimension can also be referred to as a sequence dimension, and the third dimension can also be referred to as a batch dimension.

[0118] It can be seen that the first input data includes at least one sequence in the first dimension, that is, the first dimension is the direction of each sequence of the first input data. The length of each sequence of the first input data in the first dimension is the same and equal to the length of the first input data in the first dimension, where each sequence of the first input data in the first dimension is used to represent the quantization information of one token in the first input data.

[0119] Specifically, the first input data includes at least one first sequence in the first dimension, and the expert matched with the first sequence is a second expert in the second device. It can be understood that the first sequence is one sequence of the first input data in the first dimension matched with the second expert. The first device can perform the same operation for any one first sequence of the first input data in the embodiment of the application.

[0120] Optionally, the first input data can also include at least one third sequence in the first dimension, and the expert matched with the third sequence is a first expert in the first device. It can be understood that the third sequence is one sequence of the first input data in the first dimension matched with the first expert. The first device can perform the same operation for any one third sequence of the first input data in the embodiment of the application.

[0121] In an optional embodiment, the first device can determine the expert matched with each token in the first input data and the device corresponding to the expert, that is, the first device can determine the expert corresponding to each sequence of the first input data in the first dimension. Optionally, the first device can match each sequence in the first input data with the multiple experts of the first expert layer in the hybrid expert model respectively, determine the expert matched with the sequence and the device corresponding to the expert. The specific matching rule between the sequence and the expert is not strictly limited in the embodiment of the application.

[0122] For example, the first device can calculate the matching score of one sequence in the first input data with the multiple experts of the first expert layer in the first hybrid expert model respectively. For example, the first sequence in the first input data can be regarded as a vector x, and the expert E i corresponding to a preset allocation vector w i , then the matching score s i of the first sequence with the expert E i deployed in the i-th device in the first expert layer can be expressed as follows:

[0123] wherein the number of elements of the vector x is equal to the number of elements of the preset allocation vector w i and i takes 1, 2, …, V in turn, V being the number of the plurality of experts of the first expert layer.

[0124] Afterwards, the first device can rank each expert in the first expert layer according to the matching scores from high to low, and determine the expert ranked in the first place as the expert matched with the first sequence and the device corresponding to the expert.

[0125] Optionally, the first device comprises a gating unit (or routing), which is configured to realize the function of determining the expert matched with each sequence in the first input data respectively by the first device.

[0126] It should be noted that one sequence in the first input data can also match a plurality of experts, and the embodiments of the present application do not make strict restrictions thereon. In actual applications, the number of experts matched with one sequence is usually one or two. For example, when one sequence matches two experts, the first device can rank each expert in the first expert layer according to the matching scores from high to low, and determine the experts ranked in the first two places as the experts matched with the sequence. For ease of description, the method embodiment shown in FIG. 3 will be explained and described below by taking one sequence in the first input data matching one expert as an example.

[0127] Further, after determining the expert matched with one sequence in the first input data in the first dimension and the device corresponding to the expert, the first device needs to send the sequence to the corresponding device so that the expert matched therewith can process the sequence.

[0128] Specifically, the first device and the second device each comprise a plurality of computing units, which are configured to jointly perform the computation of the experts belonging to one expert layer in the device. Any two computing units in the plurality of computing units can establish a communication connection for data transmission. That is, the first device comprises M first computing units, and the second device comprises second computing units, the M first computing units jointly perform the computation of the first expert, and the M second computing units jointly perform the computation of the second expert. It can be understood that the first device and the second device can be any two devices of the X devices shown in the embodiment of FIG. 2b.

[0129] Further, in the embodiments of the present application, the kth first computing unit in the M first computing units and the kth second computing unit in the M second computing units establish a communication connection, M being a positive integer greater than 1, and k taking 1, 2, …, M.

[0130] For ease of understanding, the case where M is equal to 4 will be further explained and described by way of example. Please refer to FIG. 5a, which is a process schematic diagram of data processing provided by the embodiments of the present application.

[0131] As shown in FIG. 5a, the first device 501 includes four first computing units, i.e., the first computing unit 5011, the first computing unit 5012, the first computing unit 5013 and the first computing unit 5014, which are respectively used to perform partial computation of the first experts in the device, and a communication connection is established between any two of the four first computing units (FIG. 5a shows the communication connection between the multiple computing units in a single device with bidirectional solid arrows). The second device 502 includes four second computing units, i.e., the second computing unit 5021, the second computing unit 5022, the second computing unit 5023 and the second computing unit 5024, which are respectively used to perform partial computation of the second experts in the device, and a communication connection is established between any two of the four second computing units.

[0132] Further, the first computing unit 5011 and the second computing unit 5021 establish a communication connection (FIG. 5a shows the communication connection between the two devices with bidirectional dashed arrows), the first computing unit 5012 and the second computing unit 5022 establish a communication connection, the first computing unit 5013 and the second computing unit 5023 establish a communication connection, and the first computing unit 5014 and the second computing unit 5024 establish a communication connection. For the explanation of the computing units, refer to the description of the corresponding part of the embodiment shown in FIG. 2b, which will not be repeated here.

[0133] Further, the first device splits the first data in the first dimension after determining the experts matched with each sequence of the first input data in the first dimension and the devices corresponding to the experts, respectively, to obtain M first sub-data.

[0134] Specifically, the length of the first input data in the first dimension is M times the length of each of the M first sub-data in the first dimension. It can be understood that each of the multiple sequences of the first input data in the first dimension is processed into M sub-sequences, and each of the M sub-sequences includes part of the data in the corresponding sequence. Then, the first device stores the M first sub-data in the M first computing units, respectively, to facilitate subsequent data transmission between devices. The number of sequences of each of the first sub-data in the first dimension is equal to the number of sequences of the first input data in the first dimension.

[0135] It can be understood that, as described above, the first input data includes a first sequence, and each of the M first sub-data includes a first sub-sequence corresponding to the first sequence. Optionally, the first input data can also include a third sequence, and each of the M first sub-data includes a third sub-sequence corresponding to the third sequence.

[0136] For example, the following takes the first input data as an example of a tensor representing the word "hello" to explain the data structure of the first sub-data. As shown in FIG. 5a, the first input data H0 is:

[0137] For specific explanations of the first input data, please refer to the description of the corresponding part of the embodiment shown in FIG. 4, which will not be repeated here.

[0138] After determining that the first sequence h1 (i.e., {0.6, 0.2, 0.2, 0.1}) matches the second expert in the second device and determining that the third sequence h3 (i.e., {0.8, 0.3, 0.2, 0.4}) matches the first expert in the first device, the first device 501 splits the first input data H0 into four first sub-data, i.e., the first sub-data H 01 , the first sub-data H 02 , the first sub-data H 03 , and the first sub-data H 04 , according to the number of first calculation units (i.e., four). It can be seen that the length of the first data in the first dimension is four times the length of each first sub-data in the first dimension.

[0139] The first sequence can be regarded as being split into four first sub-sequences, i.e., the first sub-sequence h 11 (i.e., {0.6}), the first sub-sequence h 12 (i.e., {0.2}), the first sub-sequence h 13 (i.e., {0.2}), and the first sub-sequence h 14 (i.e., {0.1}). The kth first sub-sequence includes data belonging to the first sequence in the kth first sub-data, and k takes 1, 2, 3, and 4 in turn. Similarly, the third sequence can be regarded as being split into four third sub-sequences, i.e., the third sub-sequence h 31 (i.e., {0.8}), the third sub-sequence h 32 (i.e., {0.5}), the third sub-sequence h 33 (i.e., {0.7}), and the third sub-sequence h 34 (i.e., {0.4}). The kth third sub-sequence includes data belonging to the third sequence in the kth first sub-data.

[0140] Then, the first device 501 can store the first sub-data H 01 , the first sub-data H 02 , the first sub-data H 03 , and the first sub-data H 04 in the first calculation unit 5011, the first calculation unit 5012, the first calculation unit 5013, and the first calculation unit 5014, respectively.

[0141] Step S302: The first device sends the M first sub-sequences to the second device.

[0142] Specifically, as known from step S301, the kth first computing unit of the first device can store the kth first sub-data, and the kth first sub-data includes the kth first sub-sequence. The first device can send the kth first sub-sequence to the kth second computing unit of the second device through the kth first computing unit of the first device, so as to realize that the M first computing units send the M first sub-sequences to the M second computing units.

[0143] For example, as shown in FIG. 5a, the first device 501 can send the first sub-sequence h 11 (i.e., {0.6}) to the second computing unit 5021 of the second device through the first computing unit 5011, send the first sub-sequence h 12 (i.e., {0.2}) to the second computing unit 5023 through the first computing unit 5013, send the first sub-sequence h 13 (i.e., {0.2}) to the second computing unit 5024 through the first computing unit 5014, and send the first sub-sequence h 14 (i.e., {0.1}) to the second computing unit 5025 through the first computing unit 5015.

[0144] In an optional embodiment, the communication mechanism between the kth first computing unit of the first device and the kth second computing unit of the second device is all-to-all communication, that is, the communication network between the computing units of the devices can be an all-to-all communication network. Specifically, each computing unit in the first device and the second device is provided with a data buffer, and the data buffer can be regarded as an array including a plurality of data blocks, the data blocks are used to store data, and the number of data blocks is equal to the number of computing units in the same all-to-all communication network.

[0145] Exemplarily, please refer to FIG. 5b, which is a process schematic diagram of data processing provided by an embodiment of the present application. As shown in FIG. 5b, the first computing unit 5011 in the first device 501 and the second computing unit 5021 in the second device 502 communicate through an all-to-all communication network, that is, the number of computing units in the same all-to-all communication network is 2. The data buffer of the first computing unit 5011 in the first device includes two data blocks, that is, data block 5112 and data block 5113. The data buffer of the second computing unit 5021 in the second device includes two data blocks, that is, data block 5122 and data block 5123. Based on the all-to-all communication mechanism, the first computing unit 5011 can be regarded as the first computing node, the second computing unit 5021 can be regarded as the second computing node, and the ith computing node can send the data in the jth data block to the ith data block in the jth computing node, where i is not equal to j. That is, the computing node sends the data in the data block to the computing node, and the computing node sends the data in the data block to the computing node. Optionally, the ith computing node can store the data to be stored in the ith data block.

[0146] As can be seen from FIG. 5a and FIG. 5b, the first computing unit 5011 can store the first sub-sequence h 11 in the data block 5113, so as to send the first sub-sequence h 11 to the data block 5122 of the second computing unit 5021. Optionally, the first computing unit 5011 can also store the third sub-sequence h 31 in the data block 5112.

[0147] In actual application, the first input data usually includes multiple sequences. After the first device determines the experts matched by each sequence in the first input data and the device corresponding to the expert, the intermediate data reflecting the matching result is obtained, which is beneficial to subsequent all-to-all communication between the computing units of different devices.

[0148] Exemplarily, please refer to FIG. 5c, which is a process schematic diagram of data processing provided by an embodiment of the present application. As shown in FIG. 5c, the first device 501 determines that the first input data includes four third sequences (that is, the third sequence h 3-1 , the third sequence h 3-2 , the third sequence h 3-3 , and the third sequence h 3-4 ) matched with the first device, and determines that the first input data includes four first sequences (that is, the first sequence h 1-1 , the first sequence h 1-2 , the first sequence h 1-3 , and the first sequence h 1-4After that, the intermediate data H1 is obtained. In the intermediate data H1, the four first sequences are arranged in sequence, and the four third sequences are arranged in sequence, and the first sequences are arranged after the third sequences.

[0149] Next, the first device 501 splits the intermediate data H1 into four first sub-data, i.e., the first sub-data H 01 , the first sub-data H 01 , the first sub-data H 01 , and the first sub-data H 01 , in the first dimension according to the number 4 of the first computing units.Each first sequence is split into four first sub-sequences, and each third sequence is split into four third sub-sequences. FIG. 5c shows that the gth first sequence is split into the fth first sub-sequence, and the gth third sequence is split into the fth third sub-sequence. 1-g-f 3-g-f It can be seen that each first sub-data includes four first sub-sequences and four third sub-sequences.

[0150] Finally, the first device 501 assigns the four first sub-data to the four first computing units respectively, so as to facilitate subsequent data interaction. For example, based on the all-to-all communication mechanism, the first computing unit 5011 can store the first four third sub-sequences in the data block 5112 and store the last four first sub-sequences in the data block 5113 according to the first sub-data H 01 , so as to subsequently send the data in the data block to the data block 5122 of the second computing unit 5021 in the second device 502.

[0151] It should be noted that in actual application, the number of third sequences matched with the first device in the first input data and the number of first sequences matched with the second device can not be equal. In order to maintain the consistency of the dimension of the specific model parameters of the experts, the number of sequences processed by the first experts in the first device needs to be equal to the number of sequences processed by the second experts in the second device. Before splitting the intermediate data, the first device will fill the sequences in the intermediate data to make the number of first sequences equal to the number of third sequences, and make the number of first sequences equal to the number M of the first computing units or a multiple of M.

[0152] Therefore, in the expert parallel scenario, after the first device determines that the expert matching the first sequence (representing a token) in the first input data is the second expert in the second device, the first device splits the first input data in the first dimension based on the number of first computing units in the first device, so that each first computing unit in the first device can send a first sub-sequence of the first sequence to the second computing unit in the second device that is in the same communication network as the first computing unit through the inter-device communication network, so that the second device receives all the data of the first sequence.

[0153] In some related technologies, after the first device determines the expert matching each sequence in the first input data and the corresponding device, the first device assigns a first input data to each first computing unit, and each first computing unit sends all the data of the first sequence to the second computing unit in the second device that is in the same communication network as the first computing unit through the inter-device communication network. Therefore, in the related technology, there is redundant data when the device performs cross-device communication.

[0154] In some related technologies, after the first device determines the expert matching each sequence in the first input data and the corresponding device, the first device processes the first input data in the second dimension, and a first computing unit in the first device can send all the data of a first sequence to the second computing unit in the second device that is in the same communication network as the first computing unit through the inter-device communication network.

[0155] For example, referring to FIG. 6, which is a process diagram of data processing involved in the related technology. As shown in FIG. 6, the first device 501 reorders the sequences in the intermediate data H1 to obtain the rearranged data H2. The third sub-sequence and the first sub-sequence in the rearranged data H2 are alternately arranged, and the fthfirst sub-sequence is arranged after the fththird sub-sequence. For the explanation and description of the intermediate data H1, please refer to the corresponding part of the embodiment shown in FIG. 5c, which will not be repeated here.

[0156] Then, the first device 501 evenly splits the rearranged data H2 into 4 sequence groups in the second dimension according to the number 4 of first computing units, and each sequence group includes a third sequence and a first sequence. Then, the first device 501 assigns the 4 sequence groups to the 4 first computing units one by one, so as to facilitate subsequent data interaction. For example, based on the all-to-all communication mechanism, the first computing unit 5011 can place the first sequence h 1-1 in the data block 5113, so as to subsequently send the first sequence h 1-1 to the data block 5122 of the second computing unit 5021 in the second device 502.

[0157] Therefore, in the embodiment of the present application, the first device processes the first input data in the first dimension, so that each first calculation unit transmits different partial data in the same sequence to the corresponding second calculation unit, reduces the amount of data transmitted during inter-device communication, avoids data redundancy, thereby saving the time of cross-device communication, and further improves the training efficiency and inference efficiency of the hybrid expert model. Moreover, the embodiment of the present application can also avoid sequence rearrangement processing of data when processing the first input data, simplify the data processing process, and further improve the training efficiency and inference efficiency of the hybrid expert model.

[0158] The data interaction between the plurality of calculation units in the first device is further explained below.

[0159] In an optional implementation, the first input data further includes a third sequence, and the kth first sub-data can further include a kth third sub-sequence. The first device can further process the kth third sub-sequence through the kth first calculation unit to obtain a kth fourth sub-sequence. The kth fourth sub-sequence is a sequence obtained by processing the kth third sub-sequence through the first expert.

[0160] For example, the principle of processing the kth third sub-sequence through the kth first calculation unit in the first device to obtain the kth fourth sub-sequence is explained below in combination with FIG. 5a and FIG. 7. Please refer to FIG. 7, which is a process schematic diagram of another data processing method provided by the embodiment of the present application.

[0161] As described above, the four first calculation units in the first device 501 perform part of the calculation of the first expert respectively. As shown in FIG. 5a and FIG. 7, the first device 501 first receives three third sub-sequences sent by three first calculation units other than the kth first calculation unit 5011 among the four first calculation units through the kth first calculation unit 5011, and the kth first calculation unit can store the kth third sub-sequence and M-1 third sub-sequences, i.e., M third sub-sequences. For example, the first device 501 receives the third sub-sequence h 32 (i.e., {0.5}), the third sub-sequence h 33 (i.e., {0.7}) and the third sub-sequence h 14 (i.e., {0.4}) sent by the first calculation unit 5012, the first calculation unit 5013 and the first calculation unit 5014 respectively through the first calculation unit 5011, and the first calculation unit 5011 can store the above three third sub-sequences and the third sub-sequence h 31 (i.e., {0.8}).

[0162] Then, the first device 501 concatenates the four third sub-sequences in the first dimension by the kth first computing unit to obtain a third sequence, and performs the partial calculation of the first expert on the third sequence to obtain a partial calculation result of the third sequence, i.e., a kth fifth sequence. It can be understood that the first device calculates four fifth sequences by four first computing units respectively. For example, the first device 501 concatenates the third sub-sequence h 31 (i.e., {0.8}), the third sub-sequence h 32 (i.e., {0.5}), the third sub-sequence h 33 (i.e., {0.7}) and the third sub-sequence h 14 (i.e., {0.4}) in the first dimension by the first computing unit 5011 to obtain a third sequence h3(i.e., {0.8, 0.3, 0.2, 0.4}), and performs the partial calculation in the third expert on the third sequence h3(i.e., {0.8, 0.3, 0.2, 0.4}) to obtain a fifth sequence corresponding to the third sequence h3, i.e., h5(i.e., {1.3, 0.8, 1.1, 0.6}). 51 (i.e., {1.3, 0.8, 1.1, 0.6}).

[0163] Then, the first device 501 receives the three fifth sequences sent by the other three first computing units by the kth first computing unit, and the kth first computing unit can store the fifth sequence h5and the three fifth sequences, i.e., four fifth sequences. The first device 501 adds the four fifth sequences in the first dimension to obtain a fourth sequence, which is the calculation result of the third sequence h3after being processed by the first expert. For example, the first device 501 adds the fifth sequence h 51 (i.e., {1.3, 0.8, 1.1, 0.6}), the fifth sequence h 51 (i.e., {0.5, 0.2, 0.3, 0.1}), the fifth sequence h 53 (i.e., {2.1, 2.3, 2.8, 0.2}) and the fifth sequence h 54 (i.e., {0.2, 1.5, 2.2, 0.6}) to obtain a fourth sequence h4(i.e., {4.1, 4.8, 6.4, 1.5}).

[0164] Since the third sequence h3can be regarded as a sequence obtained by concatenating four third sub-sequences in the first dimension, the first device can split the fourth sequence by the kth first computing unit to obtain a sequence obtained by processing the kth third sub-sequence by the first expert, i.e., a kth fourth sub-sequence. For example, the first device 501 splits the fourth sequence h4(i.e., {4.1, 4.8, 6.4, 1.5}) by the first computing unit 5011 to obtain a fourth sub-sequence h 41 (i.e., {4.1}).

[0165] Optionally, the communication mechanism between the M first computing units in the first device includes an all-gather communication mechanism and an all-reduce communication mechanism. For example, the operation of "receiving, by the kth first computing unit 5011, three third sub-sequences sent by three first computing units other than the kth first computing unit 5011 in the four first computing units, and splicing the four third sub-sequences in the first dimension to obtain a third sequence" can be implemented based on the all-gather communication mechanism between the four first computing units in the first device 501. The operation of "receiving, by the kth first computing unit, three fifth sequences sent by the other three first computing units, and summing the four fifth sequences in the first dimension to obtain a fourth sequence" can be implemented based on the all-reduce communication mechanism between the four first computing units in the first device.

[0166] It should be noted that FIG. 7 only illustrates the process of sending data from three first computing units other than the first first computing unit to the first first computing unit in the four first computing units. In fact, each of the four first computing units will send data to the other three first computing units.

[0167] In an optional implementation, the second device can determine that the expert matching the sixth sequence in the first dimension of the second input data of the first expert layer is the first expert in the first device.

[0168] It can be understood that the second input data of the first expert layer is the output data obtained by sequentially processing the input data of the second device by the functional layer (such as the self-attention layer) arranged in front of the first expert layer in the second device, and the input data of the second device can be regarded as a sub-input data of the input data of the mixed expert model. For the explanation and description of the second input data and the sixth sequence, please refer to the explanation and description of the first input data and the first sequence in the foregoing, which will not be repeated here. For the principle of determining that the expert matching the sixth sequence is the first expert in the first device by the second device, please refer to the principle of determining that the expert matching the first sequence is the second expert in the second device by the first device in step S301, which will not be repeated here.

[0169] Then, the second device processes the second input data to obtain M second sub-data. The kth second sub-data in the M second sub-data includes a kth sixth sub-sequence, and the kth sixth sub-sequence is the data belonging to the sixth sequence in the kth second sub-data. For the explanation and description of the second sub-data and the sixth sub-sequence, please refer to the explanation and description of the first sub-data and the first sub-sequence in the foregoing, which will not be repeated here. For the principle of processing the second input data by the second device to obtain M second sub-data, please refer to the principle of processing the first input data by the first device to obtain M first sub-data in step S301, which will not be repeated here.

[0170] Afterwards, the first device can receive the kth sixth sub-sequence sent by the kth second computing unit of the second device through the kth first computing unit of the first device, and process the kth sixth sub-sequence through the kth first computing unit to obtain a kth seventh sub-sequence, wherein the kth seventh sub-sequence is a sequence obtained by processing the kth sixth sub-sequence through the first expert. The explanation and description of the seventh sub-sequence can be referred to the explanation and description of the fourth sub-sequence, which will not be repeated here. The principle of processing the kth sixth sub-sequence through the first device to obtain the kth seventh sub-sequence can be referred to the principle of processing the kth third sub-sequence through the first device to obtain the kth fourth sub-sequence in step S302, which will not be repeated here.

[0171] Optionally, the communication mechanism between the M second computing units in the second device is the all-gather communication mechanism and the all-reduce communication mechanism, which is the same as the first device.

[0172] Step S303: The second device sends M second sub-sequences to the first device.

[0173] It can be understood that after the first device sends the M first sub-sequences to the second device, the second device processes the M first sub-sequences based on the second expert to obtain M second sub-sequences.

[0174] Specifically, after the first device sends the kth first sub-sequence to the kth second computing unit of the second device through the kth first computing unit, the second device processes the kth first sub-sequence through the kth second computing unit to obtain the kth second sub-sequence. Wherein the kth second sub-sequence is a sequence obtained by processing the kth first sub-sequence through the second expert. Afterwards, the second device sends the kth second sequence to the kth first computing unit of the first device through the kth second computing unit, so as to realize that the M first computing units send the M second sub-sequences to the M second computing units.

[0175] The explanation and description of the second sub-sequence can be referred to the explanation and description of the fourth sub-sequence, which will not be repeated here. The principle of processing the kth first sub-sequence through the second device to obtain the kth second sub-sequence can be referred to the principle of processing the kth third sub-sequence through the first device to obtain the kth fourth sub-sequence in step S302, which will not be repeated here.

[0176] Step S304: The first device obtains the first output data of the first expert layer according to the M second sub-sequences.

[0177] Specifically, after receiving the kth second sub-sequence by the kth first computing unit, the first device receives M-1 second sub-sequences sent by M-1 first computing units other than the kth first computing unit from the M first computing units by the kth first computing unit, and the kth first computing unit stores the kth second sub-sequence and the M-1 second sub-sequences, i.e., M second sub-sequences. Then, the first device obtains the first output data of the first expert layer according to the M second sub-sequences. The first output data of the first expert layer is data obtained by processing the first input data by the first expert layer, and the first output data includes a second sequence, which is a sequence obtained by processing the first sequence by the second expert.

[0178] Specifically, considering that the kth second sub-sequence is a sequence obtained by processing the kth first sub-sequence by the second expert, and the first sequence can be regarded as a sequence obtained by concatenating the M first sub-sequences in the first dimension, the first device can concatenate the M second sub-sequences in the first dimension to obtain the second sequence.

[0179] As can be seen from step S303, in an optional embodiment, the first input data can further include a third sequence, the kth first sub-data can further include a kth third sub-sequence, and the first device can further process the kth third sub-sequence by the kth first computing unit to obtain a kth fourth sub-sequence.

[0180] Then, the first device can further obtain the first output data according to the M fourth sub-sequences obtained by the M first computing units. The first output data can further include a fourth sequence, which is a sequence obtained by processing the third sequence by the first expert. Specifically, considering that the kth fourth sub-sequence is a sequence obtained by processing the kth third sub-sequence by the first expert, and the third sequence can be regarded as a sequence obtained by concatenating the M third sub-sequences in the first dimension, the first device can concatenate the M fourth sub-sequences in the first dimension to obtain the fourth sequence.

[0181] As can be seen from step S303, in an optional embodiment, the second device can determine that the expert matching the sixth sequence of the second input data of the first expert layer in the first dimension is the first expert in the first device, the first device can receive the kth sixth sub-sequence sent by the kth second computing unit of the second device by the kth first computing unit of the first device, and process the kth sixth sub-sequence by the kth first computing unit to obtain the kth seventh sub-sequence.

[0182] Then, the first device can further send, by the kth first computing unit, a kth seventh subsequence to a kth second computing unit of the second device, and the second device can obtain second output data of the first expert layer according to M seventh subsequences received by the M first computing units. The second output data of the first expert layer is data processed by the first expert layer from the second input data, and the second output data includes a seventh sequence, which is a sequence obtained by processing the sixth sequence by the first expert. For the principle of obtaining the second output data from the second device according to the M seventh subsequences, refer to the principle of obtaining the first output data from the first device according to the M second subsequences in step S304, which will not be repeated here.

[0183] In summary, in the expert parallel scenario, the M computing units of one device and the M computing units of another device in the embodiment of the application are one-to-one correspondence, each computing unit of one device only transmits data with the corresponding computing unit of another device when cross-device communication, and the M computing units of each device can transmit data when intra-device communication. In the embodiment of the application, after one device (such as the first device) determines that the expert matching the sequence (such as the first sequence) in the input data (such as the first input data) of the device is the expert in another device (such as the second device), the first input data is split in the first dimension, so that the M computing units of the device respectively send M partial data (such as M first subsequences) split from the sequence to the corresponding computing units in another device, and respectively receive M computing results (such as M second subsequences) obtained by processing M partial data sent by another device by experts, which is conducive to obtaining output data (such as first output data) subsequently.

[0184] As can be seen, compared with the mode that the M computing units of one device send all data of a sequence to the corresponding computing units in another device, in the embodiment of the application, each computing unit of one device transmits different partial data in the same sequence with the corresponding computing unit in another device, which can reduce the amount of data transmitted during cross-device communication, avoid data redundancy, save the time of cross-device communication, and further improve the training efficiency and inference efficiency of the hybrid expert model. Moreover, in the embodiment of the application, each device can avoid sequence rearrangement processing when processing the input data of the device, simplifying the data processing process and further improving the training efficiency and inference efficiency of the hybrid expert model.

[0185] In addition, the plurality of computing units of each device in the embodiments of the present application can also splice the partial data of a sequence (such as the second sub-sequence) matched with the experts in the device to obtain all data (such as the second sequence) of the sequence, facilitating subsequent processing. Compared with the mode that the M computing units of one device send all data of one sequence to the other device through the communication network between devices, the embodiments of the present application can restore the split sub-sequences to complete sequences through the intra-device communication process with a higher communication rate, and reduce the amount of data processed in the inter-device communication process with a lower communication rate, which not only can improve the training efficiency and inference efficiency of the hybrid expert model, but also can enhance the accuracy of the training result and inference result of the hybrid expert model.

[0186] In an optional implementation, the embodiments of the present application can also split the first input data into a plurality of first data, and input the plurality of first data into different execution data streams respectively, so as to save the total data processing time of a single expert layer, and further improve the training efficiency and inference efficiency of the hybrid expert model.

[0187] Specifically, please refer to FIG. 8, which is a flowchart of another data processing method provided by the embodiments of the present application.

[0188] Step S801: The first device processes the first input data to obtain N first data.

[0189] Specifically, for any one first data, the length of the first input data in the first dimension is equal to the length of the first data in the first dimension, the length of the first input data in the second dimension is W1 times the length of the first data in the second dimension, the length of the first input data in the third dimension is W2 times the length of the first data in the third dimension, N is equal to the product of W1 and W2, W1 is a positive integer, and W2 is a positive integer greater than or equal to 2.

[0190] Specifically, please refer to FIG. 9a, which is a process diagram of another data processing provided by the embodiments of the present application. The first input data shown in FIG. 9a is the first input data H0 shown in FIG. 4. The first device can split and process the first data in the third dimension to obtain two first data, i.e., the first data H 11 and the first data H 12 Based on the description of the embodiment shown in FIG. 4, it can be known that the lengths of the first input data in the first dimension, the second dimension and the third dimension are 6, 6 and 6 respectively, and the lengths of each of the two first data in the first dimension, the second dimension and the third dimension are 6, 6 and 3 respectively.

[0191] For example, please refer to FIG. 9b, which is a schematic diagram of another data processing process according to an embodiment of the present application. The first input data shown in FIG. 9b is the first input data H0 shown in FIG. 4. The first device can sequentially split the first data in the third dimension and the second dimension to obtain four first data, i.e., the first data H 11 , the first data H 12 , the first data H 13 , and the first data H 14 . Based on the description of the embodiment shown in FIG. 4, the length of the first input data in the first dimension, the second dimension, and the third dimension is 6, 6, and 6 respectively, and the length of each of the four first data in the first dimension, the second dimension, and the third dimension is 6, 3, and 3 respectively. It should be noted that FIG. 9a and FIG. 9b both show the splitting manner of the first input data in different gray scales.

[0192] It can be understood that the first device can split the first input data into N first data averagely.

[0193] Step S802: The first device sequentially starts processing the N first data according to the first time offset to obtain the second data.

[0194] Since the length of the first input data in the first dimension is equal to the length of the first data in the first dimension, and the sequence of the first data in the first dimension is used to represent the quantization information of a token in the first input data, the first device can perform the operation on each first data as the first device performs on the first input data in the embodiment shown in FIG. 3, i.e., the operation performed by the first device on the first input data in steps S301 to S304.

[0195] That is, the first device can perform the following steps on each first data:

[0196] Step one: The first device splits the first data in the first dimension to obtain M first sub-data.

[0197] Step two: The first device sends the M first sub-sequences to the second device.

[0198] Step three: The first device receives the M second sub-sequences sent by the second device.

[0199] Step four: The first device obtains the second data according to the M second sub-sequences.

[0200] The principle of the first device and the second device in steps one to four is described in the corresponding part of steps S301 to S304 in the embodiment shown in FIG. 3, which will not be repeated here.

[0201] It can be understood that the data processing procedure of the single expert layer in the embodiments of the present application can be sequentially summarized as five procedures in time sequence: the first procedure is the procedure of transmitting the sub-sequences across devices, the second procedure is the procedure of splicing the sub-sequences in the same device, the third procedure is the procedure of performing expert calculation by the calculation units in the same device, the fourth procedure is the procedure of transmitting the sub-sequences across devices, and the fifth procedure is the procedure of splicing the sub-sequences in the same device.

[0202] For the embodiment shown in FIG. 3, for example, in the first procedure, the first device can send the kth first sub-sequence to the kth second calculation unit of the second device through the kth first calculation unit; in the second procedure, the second device can receive M-1 first sub-sequences sent by M-1 second calculation units through the kth second calculation unit, and splice the four first sub-sequences in the first dimension to obtain a first sequence; in the third procedure, the second device processes the third sequence through the kth first calculation unit to obtain a kth second sub-sequence; in the fourth procedure, the second device sends the kth second sub-sequence to the kth first calculation unit of the first device through the kth second calculation unit; and in the fifth procedure, the first device can receive M-1 second sub-sequences sent by M-1 first calculation units through the kth first calculation unit, and splice the four second sub-sequences in the first dimension to obtain a second sequence.

[0203] In addition, as known from the foregoing, the communication network between the calculation units of the devices in the first procedure and the fourth procedure can be an all-to-all communication network, and the communication network between the calculation units in the device in the second procedure and the fifth procedure can be an all-gather communication network. For ease of description, the data processing time of the first procedure and the data processing time of the fourth procedure are referred to as the first all-to-all communication time and the second all-to-all communication time, respectively, the data processing time of the second procedure and the data processing time of the fifth procedure are referred to as the first all-gather communication time and the second all-gather communication time, respectively, and the data processing time of the third procedure is referred to as the expert calculation time.

[0204] For example, as shown in FIGS. 9a and 9b, the total data processing time T0 (i.e., the total data processing time of the first input data) of the embodiment shown in FIG. 3 can be considered to be equal to the first all-to-all communication time T a1 corresponding to the first input data H0, the first all-gather communication time T g1 corresponding to the first input data H0, the expert calculation time T e corresponding to the first input data H0, and the second all-to-all communication time T a2a second all-gather communication time T corresponding to the first input data H0 g2 . It can be understood that T a1 is equal to T a2 , T g1 is equal to T g2 . Due to the low cross-device communication rate, T a1 is usually greater than T g1 .

[0205] In the embodiment shown in FIG. 8, the total data processing time T' of each first data can be considered as a sum of the first all-to-all communication time T' corresponding to the first data, the first all-gather communication time T' corresponding to the first data, the expert calculation time T' corresponding to the first data H0, the second all-to-all communication time T' corresponding to the first data, and the second all-gather communication time T' corresponding to the first data. H a1 g1 e a2 g2 It can be understood that T' is equal to T' a1 , T' is equal to T' a2 , T' is greater than T' g1 , T' is less than T a2 , T' is less than T a1 , T' is less than T g1 , T' is less than T a1 , T' is less than T a1 , T' is less than T g1 , T' is less than T g1 , and T' is less than T e . e

[0206] Considering that the communication networks used for cross-device communication and intra-device communication can be different, for example, the all-to-all communication network can be a remote direct memory access (RDMA) network, and the all-gather communication network can be a cache coherence (HCCS) network, i.e., there is no bandwidth competition in hierarchical communication. Therefore, there is no bandwidth competition between the two adjacent processes in the five processes described above.

[0207] The first device can input the N first data into a plurality of execution data streams respectively, each execution data stream corresponding to an execution time, and the time difference between adjacent execution data streams being a first time offset. The first device can start executing steps 1-5 on the first data according to the execution time corresponding to each first data. That is, the first device can start processing the N first data in turn according to the first time offset to obtain the second data. The first time offset is the first all-to-all communication time T'​​​​​​a1 .

[0208] As shown in FIG. 9a, the first device first processes the first data H 11 (which can be regarded as a first data stream) and starts to process the first data H a1 (which can be regarded as a second data stream) after an interval T' 12 It can be seen that, since there is no wideband contention between the two adjacent processes, the first device can execute the operation in the second process (i.e., execute the operation related to all-gather communication) while synchronously executing the operation in the first process (i.e., execute the operation related to all-to-all communication). The first device can also execute the operation in the third process (i.e., execute the operation related to expert calculation) while synchronously executing the operation in the second process. 11 12 11 12

[0209] Compared with the total data processing time T0 of the embodiment shown in FIG. 3, the total data processing time T0' of the embodiment shown in FIG. 8 can be calculated by the following formula:

[0210] T0' = 2T' a1 + T' g1 + T e ' + T' a2 + T' g2

[0211] Similarly, as shown in FIG. 9b, the first device first processes the first data H 11 (which can be regarded as a first data stream) and starts to process the first data H a1 , the first data H 12 , and the first data H 13 in turn at intervals T' 14 .

[0212] It can be understood that the total data processing time T0' of the embodiment shown in FIG. 8 is less than the total data processing time T0 of the embodiment shown in FIG. 3. It should be noted that the number of the first data in the embodiments of the present application is not strictly limited, that is, the embodiments of the present application can freely schedule the granularity of the data stream according to the needs of actual application scenarios.

[0213] ​​​​Therefore, each device in the embodiments of the present application can also split the input data of the device in the third dimension and / or the second dimension, so as to realize concurrent execution of the communication process and the calculation process between different data streams through fine-grained scheduling, thereby masking the communication time, shortening the end-to-end execution time, and further improving the training efficiency and inference efficiency of the hybrid expert model.

[0214] The data processing apparatus provided by the embodiments of the present application will be introduced below.

[0215] FIG. 10 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application. As shown in FIG. 10, the data processing apparatus 100 includes a processing module 1001 and a transceiver module 1002. The transceiver module 1002 can realize corresponding communication functions, and the processing module 1001 is configured to perform data processing. The transceiver module 1002 can also be referred to as an interface, a communication interface, or a communication module, etc.

[0216] In some embodiments of the present application, the data processing apparatus can be used to perform the actions performed by the first device or the second device in the above method embodiments. The transceiver module 1002 is configured to perform the transceiving operations performed by the first device or the second device in the above method embodiments, such as the sending end being the device itself or a chip or a functional module configured in the device, etc. The processing module 1001 is configured to perform the processing operations performed by the first device or the second device in the above method embodiments. The processing module 1001 can perform corresponding operations by calling a computer program, or can perform corresponding operations by using a corresponding hardware circuit. The transceiver module 1002 can independently perform transceiving operations, or can perform corresponding transceiving operations under the control of the processing module 1001.

[0217] For example, the data processing apparatus 100 shown in FIG. 10 can be the first device or a device in the first device in the above method embodiments. The data processing apparatus 100 includes a first expert, and the first expert is an expert in a first expert layer in a hybrid expert model. The first expert layer includes the first expert and a second expert. The processing module 1001 and the transceiver module 1002 in the data processing apparatus 100 can perform the following operations respectively:

[0218] The processing module 1001 is configured to split the first data in the first dimension to obtain M first sub-data, wherein the first data includes a plurality of sequences, each sequence in the plurality of sequences corresponds to an expert, and each sequence is configured to represent quantization information of a token in the first data, and the first dimension is the direction of each sequence.

[0219] The transceiver module 1002 is configured to send M first sub-sequences to the second device, wherein each of the M first sub-data comprises a first sub-sequence, each of the M first sub-sequences comprises partial data of the first sequence, the first sequence is one of the plurality of sequences in the first data, and the first sequence corresponds to the second expert of the second device.

[0220] The transceiver module 1002 is configured to receive M second sub-sequences sent by the second device, wherein the M second sub-sequences are sequences obtained by processing the M first sub-sequences by the second expert respectively.

[0221] The processing module 1001 is configured to obtain the second data according to the M second sub-sequences, wherein the second data is data obtained by processing the first data by the first expert and / or the second expert, and the second sequence of the second data in the first dimension is a sequence obtained by processing the first sequence by the second expert.

[0222] In an optional implementation, the lengths of the M first sub-sequences are equal, and in terms of sending the M first sub-sequences to the second device:

[0223] The transceiver module 1002 is specifically configured to send the kth first sub-sequence to the kth second computing unit of the second device through the kth first computing unit of the first device, so as to realize that the M first computing units send the M first sub-sequences to the M second computing units, wherein the kth first sub-sequence comprises data belonging to the first sequence in the kth first sub-data of the M first sub-data.

[0224] In terms of receiving the M second sub-sequences sent by the second device:

[0225] The transceiver module 1002 is specifically configured to receive the kth second sub-sequence sent by the kth second computing unit through the kth first computing unit, so as to realize that the M first computing units receive the M second sub-sequences sent by the M second computing units, wherein the kth second sub-sequence is a sequence obtained by processing the kth first sub-sequence by the second expert, and k is an integer from 1 to M in turn.

[0226] In an optional implementation, the second sequence is a sequence obtained by splicing the M second sub-sequences in the first dimension.

[0227] In an optional implementation, in terms of obtaining the second data according to the M first processing results received by the M first computing units:

[0228] The transceiver module 1002 is configured to receive M-1 second sub-sequences sent by M-1 first computing units other than the kth first computing unit among the M first computing units by the kth first computing unit.

[0229] The processing module 1001 is configured to splice the received M second subsequences in the first dimension by the kth first calculation unit to obtain a second sequence.

[0230] The processing module 1001 is configured to obtain second data according to the second sequence.

[0231] In an optional implementation, in the aspect of obtaining the second data according to the M first processing results received by the M first calculation units, the processing module 1001 is specifically configured to:

[0232] receive M-1 second subsequences sent by M-1 first calculation units other than the kth first calculation unit among the M first calculation units through the kth first calculation unit;

[0233] splice the received M second subsequences in the first dimension by the kth first calculation unit to obtain a second sequence;

[0234] obtain second data according to the second sequence.

[0235] In an optional implementation, before the first data is split in the first dimension, the processing module 1001 is further configured to determine that the expert matched with the first sequence is a second expert.

[0236] In an optional implementation, the processing module 1001 is further configured to:

[0237] determine that the expert matched with the third sequence is a first expert, wherein the third sequence is one sequence in the plurality of sequences in the first data, the third sequence corresponds to the first expert, and the M first sub-data respectively include M third subsequences, and each third subsequence in the M third subsequences includes partial data of the third sequence;

[0238] process the kth third subsequence by the kth first calculation unit to obtain a kth fourth subsequence, wherein the kth fourth subsequence is a sequence obtained by processing the kth third subsequence by the first expert;

[0239] obtain the second data according to the M second subsequences received by the M first calculation units, including:

[0240] obtain the second data according to the M second subsequences and the M fourth subsequences, wherein a fourth sequence of the second data in the first dimension is a sequence obtained by processing the third sequence by the first expert.

[0241] In an optional implementation, the fourth sequence is a sequence obtained by splicing the M fourth subsequences in the first dimension.

[0242] In an optional implementation, in the processing of the kth third subsequence by the kth first computing unit to obtain the kth fourth subsequence, the processing module 1001 is specifically configured to:

[0243] The kth first computing unit receives M-1 third subsequences sent by M-1 first computing units, where the M-1 first computing units are computing units other than the kth first computing unit in the M first computing units;

[0244] The kth first computing unit processes the third sequence to obtain the kth fifth sequence, where the third sequence is a sequence obtained by concatenating the kth third subsequence and the M-1 third subsequences in the first dimension;

[0245] The kth first computing unit receives M-1 fifth sequences sent by M-1 first computing units;

[0246] The kth first computing unit obtains the kth fourth subsequence according to the fourth sequence, where the fourth sequence is a sequence obtained by summing the M fifth sequences.

[0247] In an optional implementation, the first data includes a plurality of sequences in the first dimension, and the third sequence is one of the plurality of sequences.

[0248] In an optional implementation,

[0249] The transceiver module 1002 is further configured to receive, by the kth first computing unit, the kth sixth subsequence sent by the kth second computing unit, to implement that the M first computing units receive M sixth subsequences sent by the M second computing units, where the M sixth subsequences have equal lengths, the kth sixth subsequence includes data belonging to the sixth sequence in the kth second subdata of the M second data, the M second data is obtained by the second device splitting the third data in the first dimension, the third data includes a plurality of sequences, each sequence of the plurality of sequences corresponds to an expert, each sequence of the third sequence is used to represent quantization information of a token in the third data, the sixth sequence is one of the plurality of sequences in the third data, and the sixth sequence corresponds to the first expert;

[0250] The processing module 1001 is further configured to process, by the kth first computing unit, the kth sixth subsequence to obtain the kth seventh subsequence, where the kth sixth subsequence includes data belonging to the sixth sequence in the kth second subdata of the M second data, and the kth seventh subsequence is a sequence obtained by processing the kth sixth subsequence by the first expert;

[0251] The transceiver module 1002 is further configured to send, by the kth first computing unit, the kth seventh subsequence to the kth second computing unit.

[0252] In an optional implementation, the communication mechanism between the kth first computing unit in the first device and the kth second computing unit in the second device is an all-to-all communication, and the communication mechanism between the M first computing units in the first device comprises an all-gather communication mechanism and an all-reduce communication.

[0253] In an optional implementation, the communication network of the all-to-all communication mechanism is a remote direct memory access (RDMA) network, and the communication network of the all-reduce communication mechanism is a cache coherence (HCCS) network.

[0254] In an optional implementation, the first input data comprises a tensor used to represent association information between a plurality of tokens in each token set of a plurality of token sets, the dimension of the first input data comprises a first dimension, a second dimension and a third dimension, the length of the first input data on the first dimension is equal to the number of all values used to quantify one token in the plurality of tokens, the length of the first input data on the second dimension is equal to the number of the plurality of tokens, and the third length of the first input data on the third dimension is equal to the number of the plurality of token sets.

[0255] In an optional implementation, the first data is the first input data.

[0256] In an optional implementation, the first data is any one of N first data obtained by splitting the first input data, and N is a positive integer greater than or equal to 2.

[0257] In an optional implementation, the processing module 1001 is further configured to sequentially start processing the N first data to obtain the second data according to a first time offset, and the first time offset is the time for a single first data to execute the step of sending the kth first sub-sequence operation from the kth first computing unit of the first device to the kth second computing unit of the second device.

[0258] In an optional implementation, the length of the first input data on the first dimension is equal to the length of the first data on the first dimension, the length of the first input data on the second dimension is W1 times the length of the first data on the second dimension, the length of the first input data on the third dimension is W2 times the length of the first data on the third dimension, N is equal to the product of W1 and W2, W1 is a positive integer, and W2 is a positive integer greater than or equal to 2.

[0259] In an optional implementation, the hybrid expert model comprises at least one expert layer and at least one feedforward neural network (FFN) layer, the at least one expert layer comprises a first expert layer, and each expert layer in the at least one expert layer comprises at least one expert.

[0260] In an optional implementation, the kth first computing unit is one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU).

[0261] In a possible implementation, in the data processing apparatus 100 shown in FIG. 10, the processing module 1001 can be one or more processors, and the transceiver module 1002 can be a transceiver, or the transceiver module 1002 can also be a sending module and a receiving module, the sending module can be a transmitter, and the receiving module can be a receiver, and the sending module and the receiving module are integrated into one device, for example, a transceiver. In the embodiments of the present application, the processor and the transceiver can be coupled, and the connection mode of the processor and the transceiver is not limited in the embodiments of the present application. In the process of executing the above method, the process of sending information in the above method can be the process of outputting the above information by the processor. When outputting the above information, the processor outputs the above information to the transceiver for transmission by the transceiver. After the above information is output by the processor, it can also need to be processed further before reaching the transceiver. Similarly, the process of receiving information in the above method can be the process of receiving inputted above information by the processor. When the processor receives the inputted information, the transceiver receives the above information and inputs it to the processor. Furthermore, after the transceiver receives the above information, the above information can need to be processed further before being inputted to the processor.

[0262] As shown in FIG. 11, the data processing apparatus 110 comprises one or more processors 1102 and a transceiver 1101. For example, the transceiver 1101 is configured to perform functions or steps implemented by the transceiver module 1002 shown in FIG. 10, and the processor 1102 is configured to perform functions or steps implemented by the processing module 1001 shown in FIG. 10. For specific descriptions of the processor 1102 and the transceiver 1101, reference can be made to the method embodiments shown in FIG. 10 or the above description, which will not be repeated here.

[0263] In the above embodiments, the related steps and information can be described with reference to the above method embodiments, which will not be repeated here.

[0264] In the various implementations of the data processing apparatus shown in FIG. 11, the transceiver can comprise a receiver configured to perform the function (or operation) of receiving and a transmitter configured to perform the function (or operation) of transmitting. The transceiver is configured to communicate with other devices / apparatuses through a transmission medium.

[0265] Optionally, the data processing device 110 can further include one or more memories 1103 for storing program instructions and / or data. The memory 1103 is coupled to the processor 1102. The coupling in the embodiments of the present application is indirect coupling or communication connection between devices, units or modules, which can be electrical, mechanical or other forms, for information interaction between devices, units or modules. The processor 1102 can operate in cooperation with the memory 1103. The processor 1102 can execute the program instructions stored in the memory 1103. Optionally, at least one of the one or more memories can be included in the processor.

[0266] The embodiments of the present application do not limit the specific connection medium between the above-mentioned transceiver 1101, processor 1102 and memory 1103. In the embodiments of the present application, the memory 1103, processor 1102 and transceiver 1101 are connected through a bus 1104 in FIG. 11, and the bus is represented by a thick line in FIG. 11, and the connection mode between other components is only schematically illustrated and is not limited. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, only one thick line is used in FIG. 11, but it does not mean that there is only one bus or only one type of bus.

[0267] In the embodiments of the present application, the processor can be a general processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor, etc.

[0268] The memory in the embodiments of the present application can include, but is not limited to, a non-volatile memory such as a hard disk drive (HDD) or a solid-state drive (SSD), a random access memory (RAM), an erasable programmable ROM (EPROM), a read-only memory (ROM), a compact disc read-only memory (CD-ROM), and the like. The memory is any storage medium that can be used to carry or store program codes in the form of instructions or data structures and can be read and / or written by a computer (such as the data processing apparatus shown in the present application and the like). The memory in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0269] The processor 1102 is mainly used for processing communication protocols and communication data, controlling the entire data processing apparatus, executing software programs, and processing data of the software programs. The memory 1103 is mainly used for storing software programs and data. The transceiver 1101 can include a control circuit and an antenna, and the control circuit is mainly used for converting baseband signals and radio frequency signals and processing radio frequency signals. The antenna is mainly used for receiving and transmitting radio frequency signals in the form of electromagnetic waves. Input and output devices, such as touch screens, display screens, keyboards, and the like, are mainly used for receiving user input data and outputting data to users.

[0270] When the data processing apparatus is powered on, the processor 1102 can read the software program in the memory 1103, interpret and execute the instructions of the software program, and process the data of the software program. When data needs to be transmitted wirelessly, the processor 1102 performs baseband processing on the data to be transmitted, and outputs the baseband signal to the radio frequency circuit. The radio frequency circuit converts the baseband signal into a radio frequency signal, and transmits the radio frequency signal in the form of electromagnetic waves through the antenna. When data is transmitted to the data processing apparatus, the radio frequency circuit receives the radio frequency signal through the antenna, converts the radio frequency signal into a baseband signal, and outputs the baseband signal to the processor 1102. The processor 1102 converts the baseband signal into data and processes the data.

[0271] In another implementation, the radio frequency circuit and the antenna can be arranged independently of the processor performing baseband processing, for example, in a distributed scenario, the radio frequency circuit and the antenna can be arranged remotely from the data processing apparatus.

[0272] The data processing apparatus shown in the embodiments of the present application can also have more components than those shown in FIG. 11, and the embodiments of the present application do not limit the same. The methods performed by the processor and the transceiver shown above are only examples, and the steps actually performed by the processor and the transceiver can refer to the methods described above.

[0273] In another possible implementation, in the data processing apparatus shown in FIG. 10, the processing module 1001 can be one or more logic circuits, and the transceiving module 1002 can be an input / output interface, also referred to as a communication interface, or an interface circuit, or an interface, etc. Alternatively, the transceiving module 1002 can also be a sending module and a receiving module, the sending module can be an output interface, and the receiving module can be an input interface. The sending module and the receiving module are integrated in one module, for example, an input / output interface. As shown in FIG. 12, the data processing apparatus 120 shown in FIG. 12 includes a logic circuit 1201 and an interface 1202. That is, the processing module 1001 described above can be implemented by the logic circuit 1201, and the transceiving module 1002 can be implemented by the interface 1202. The logic circuit 1201 can be a chip, a processing circuit, an integrated circuit, or a system on chip (SoC) chip, etc., and the interface 1202 can be a communication interface, an input / output interface, a pin, etc. For example, FIG. 12 is a chip including the logic circuit 1201 and the interface 1202, which is shown by taking the data processing apparatus described above as an example.

[0274] In the embodiments of the present application, the logic circuit and the interface can also be coupled to each other. The embodiments of the present application do not limit the specific connection mode of the logic circuit and the interface. For example, the logic circuit 1201 can be used to perform the functions or steps implemented by the processing module 1001 shown in FIG. 10, and the interface 1202 can be used to perform the functions or steps implemented by the transceiving module 1002 shown in FIG. 10. For specific descriptions of the logic circuit 1201 and the interface 1202, refer to the method embodiments shown above or FIG. 10 or FIG. 11, which will not be described here in detail.

[0275] The above description of the data processing apparatus is only an example, and the specific description of the data processing apparatus shown in FIG. 12 can also refer to the method embodiments above or FIG. 10 or FIG. 11, which will not be described here in detail.

[0276] The data processing apparatus shown in the embodiments of the present application can implement the methods provided by the embodiments of the present application in the form of hardware, or implement the methods provided by the embodiments of the present application in the form of software, etc., and the embodiments of the present application do not limit the same.

[0277] In the above various embodiments, the relevant steps and information can be referred to the description in the method embodiments, which will not be repeated here. For the specific implementation of each embodiment shown in FIG. 12, the above various embodiments can also be referred to, which will not be repeated here.

[0278] The embodiments of the present application further provide a data processing system, which comprises the first device and the second device in any of the above method embodiments.

[0279] In addition, the present application further provides a computer program for implementing the operations and / or processes performed by the first device in the method provided by the present application.

[0280] The present application further provides a computer readable storage medium, which stores computer code, when the computer code is run on a computer, the computer code causes the computer to perform the operations and / or processes performed by the various data processing devices in the method provided by the present application.

[0281] The present application further provides a computer program product, which comprises computer code or a computer program, when the computer code or the computer program is run on a computer, the operations and / or processes performed by the first device in the method provided by the present application are performed.

[0282] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connection.

[0283] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, that is, they can be located in one place, or can be distributed on a plurality of network modules. Some or all of the modules can be selected according to actual needs to achieve the technical effects of the scheme provided by the embodiments of the present application.

[0284] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically, or two or more modules can be integrated into one module. The above integrated module can be realized in the form of hardware or software functional module.

[0285] The integrated module, if implemented in the form of a software function module and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, including a number of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned readable storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0286] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described as "exemplary" or "for example" in the present application should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0287] In addition, unless otherwise stated, the ordinal numbers "first", "second", etc. used in the embodiments of the present application are used to distinguish multiple objects, and are not intended to limit the order, time sequence, priority, or importance of the multiple objects, such as the first device and the first sequence.

[0288] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized by, The method is applied to a first device, the first device comprising a first expert, and the method comprises: splitting first data in a first dimension to obtain M first sub-data, wherein the first data comprises a plurality of sequences, each sequence of the plurality of sequences corresponds to an expert, and each sequence is used to represent quantization information of a token in the first data, and the first dimension is a direction of each sequence; sending M first sub-sequences to a second device, wherein each first sub-data of the M first sub-data comprises a first sub-sequence, each first sub-sequence of the M first sub-sequences comprises partial data of a first sequence, the first sequence is one of the plurality of sequences in the first data, and the first sequence corresponds to a second expert of the second device; receiving M second sub-sequences sent by the second device, wherein the M second sub-sequences are sequences obtained by processing the M first sub-sequences by the second expert respectively; obtaining second data according to the M second sub-sequences, wherein the second data is data obtained by processing the first data by the first expert and / or the second expert, and a second sequence of the second data in the first dimension is a sequence obtained by processing the first sequence by the second expert.

2. The method of claim 1, wherein, The lengths of the M first sub-sequences are equal, and the sending of the M first sub-sequences to the second device comprises: sending a kth first sub-sequence to a kth second computing unit of the second device by a kth first computing unit of the first device, so as to realize the sending of the M first sub-sequences from the M first computing units of the first device to the M second computing units of the second device; The receiving of the M second sub-sequences sent by the second device comprises: receiving a kth second sub-sequence sent by the kth second computing unit by the kth first computing unit, so as to realize the receiving of the M second sub-sequences sent by the M second computing units by the M first computing units, wherein the k takes an integer from 1 to M in turn.

3. The method of claim 2, wherein: the second sequence is a sequence obtained by splicing the M second sub-sequences in the first dimension.

4. The method of claim 3, wherein, The obtaining of the second data according to the M second sub-sequences comprises: receiving M-1 second sub-sequences sent by M-1 first computing units other than the kth first computing unit among the M first computing units by the kth first computing unit; splicing the received M second sub-sequences in the first dimension by the kth first computing unit to obtain the second sequence; obtaining the second data according to the M second sub-sequences.

5. The method according to any one of claims 2 to 4, characterized in that, Before the splitting of the first data in the first dimension, the method further comprises: determining that the expert matched with the first sequence is the second expert.

6. The method according to any one of claims 2-5, characterized in that, The method further comprises: determining that the expert matching the third sequence is the first expert, wherein the third sequence is one of a plurality of sequences in the first data, the third sequence corresponds to the first expert, the M first sub-data respectively include M third sub-sequences, and each of the M third sub-sequences includes partial data of the third sequence; processing, by the kth first computing unit, the kth third sub-sequence to obtain a kth fourth sub-sequence, wherein the kth fourth sub-sequence is a sequence obtained by processing the kth third sub-sequence by the first expert; the second data is obtained according to the M second sub-sequences, including: the second data is obtained according to the M second sub-sequences and the M fourth sub-sequences, wherein a fourth sequence of the second data in the first dimension is a sequence obtained by processing the third sequence by the first expert.

7. The method of claim 6, wherein: the fourth sequence is a sequence obtained by splicing the M fourth sub-sequences in the first dimension.

8. The method of claim 7, wherein, the processing, by the kth first computing unit, of the kth third sub-sequence to obtain a kth fourth sub-sequence, includes: receiving, by the kth first computing unit, M-1 third sub-sequences sent by M-1 first computing units, wherein the M-1 first computing units are computing units other than the kth first computing unit among the M first computing units; processing, by the kth first computing unit, the third sequence to obtain a kth fifth sequence, wherein the third sequence is a sequence obtained by splicing the kth third sub-sequence and the M-1 third sub-sequences in the first dimension; receiving, by the kth first computing unit, M-1 fifth sequences sent by the M-1 first computing units; obtaining, by the kth first computing unit, the kth fourth sub-sequence according to a fourth sequence, wherein the fourth sequence is a sequence obtained by summing the M fifth sequences.

9. The method according to any one of claims 5-8, characterized in that, the first data includes a plurality of sequences in the first dimension, and the third sequence is one of the plurality of sequences.

10. The method according to any one of claims 2-9, characterized in that, The method further includes: receiving, by the kth first computing unit, a kth sixth sub-sequence sent by the kth second computing unit to achieve that the M first computing units receive M sixth sub-sequences sent by the M second computing units, wherein lengths of the M sixth sub-sequences are equal, the kth sixth sub-sequence includes data belonging to a sixth sequence in a kth second sub-data of M second sub-data, the M second sub-data is obtained by the second device by slicing third data in the first dimension, the third data includes a plurality of sequences, each sequence of the plurality of sequences corresponds to an expert, each sequence of the third sequence is used to represent quantization information of a token in the third data, and the sixth sequence is one of the plurality of sequences in the third data, and the sixth sequence corresponds to the first expert. The kth first computing unit processes the kth sixth sub-sequence to obtain a kth seventh sub-sequence, wherein the kth seventh sub-sequence is a sequence obtained by processing the kth sixth sub-sequence by the first expert; The kth first computing unit sends the kth seventh sub-sequence to the kth second computing unit.

11. The method of any one of claims 2-10, wherein: The communication mechanism between the kth first computing unit in the first device and the kth second computing unit in the second device is an all-to-all communication, and the communication mechanism between the M first computing units in the first device includes an all-gather communication mechanism and an all-reduce communication.

12. The method of claim 11, wherein, The communication network of the all-to-all communication mechanism is a remote direct memory access (RDMA) network, and the communication network of the all-reduce communication mechanism is a cache coherence (HCCS) network.

13. The method according to any one of claims 1 to 12, characterized in that, The first expert is an expert in a first expert layer of a hybrid expert model, the first expert layer includes the first expert and the second expert, the first data is first input data of the first expert layer, the first input data includes a tensor for representing association information between a plurality of tokens in each token set, the dimensions of the first input data include the first dimension, a second dimension, and a third dimension, the length of the first input data in the first dimension is equal to the number of all values used to quantify a token in the plurality of tokens, the length of the first input data in the second dimension is equal to the number of the plurality of tokens, and the third length of the first input data in the third dimension is equal to the number of the plurality of token sets.

14. The method of claim 13, wherein, The first data is the first input data.

15. The method of claim 13, wherein, The first data is any one of N first data obtained by splitting the first input data, and N is a positive integer greater than or equal to 2.

16. The method of claim 15, wherein, The first device sequentially starts processing the N first data to obtain the second data according to a first time offset, and the first time offset is the time for performing the operation of sending the kth first sub-sequence from the kth first computing unit of the first device to the kth second computing unit of the second device for a single first data.

17. The method according to claims 15 and 16, characterized in that, The length of the first input data in the first dimension is equal to the length of the first data in the first dimension, the length of the first input data in the second dimension is W1 times the length of the first data in the second dimension, the length of the first input data in the third dimension is W2 times the length of the first data in the third dimension, N is equal to the product of W1 and W2, W1 is a positive integer, and W2 is a positive integer greater than or equal to 2.

18. The method according to any one of claims 13-17, characterized by, The mixed expert model comprises at least one expert layer and at least one feedforward neural network (FFN) layer, the at least one expert layer comprises the first expert layer, and each expert layer in the at least one expert layer comprises at least one expert.

19. The method of any one of claims 2-18, wherein, The kth first computing unit is one of a central processing unit (CPU), a graphics processing unit (GPU) and a neural network processing unit (NPU).

20. A data processing apparatus, characterized in that, The data processing apparatus comprises a processor, a memory and a transceiver, the transceiver is configured to perform receiving and / or transmitting operations under control of the processor, the memory is configured to store a computer program, and the processor is configured to invoke the computer program, wherein: The processor is configured to split the first data in a first dimension to obtain M first sub-data, wherein the first data comprises a plurality of sequences, each sequence in the plurality of sequences corresponds to an expert, and each sequence is used to represent quantization information of a token in the first data, and the first dimension is a direction of each sequence; The transceiver is configured to send M first sub-sequences to a second device, wherein each first sub-data in the M first sub-data comprises a first sub-sequence, each first sub-sequence in the M first sub-sequences comprises partial data of a first sequence, the first sequence is one sequence in the plurality of sequences in the first data, and the first sequence corresponds to a second expert of the second device; The transceiver is configured to receive M second sub-sequences sent by the second device, wherein the M second sub-sequences are sequences obtained by processing the M first sub-sequences by the second expert respectively; The processor is configured to obtain second data according to the M second sub-sequences, wherein the second data is data obtained by processing the first data by the first expert and / or the second expert, and a second sequence of the second data in the first dimension is a sequence obtained by processing the first sequence by the second expert.

21. A data processing apparatus, characterized in that, The data processing apparatus comprises a module for performing the method of any one of claims 1-19.

22. A data processing apparatus, characterized in that, The logic circuit and the interface are coupled; The interface is configured to input and / or output information, and the logic circuit is configured to perform the method of any one of claims 1-19.

23. A communication system, characterized by The communication system comprises a first device and a second device, the first device is the first device of any one of claims 1-19, and the second device is the second device of any one of claims 1-19.

24. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store a computer program, and the computer program is configured to perform the method of any one of claims 1-19 when executed.

Images