Data processing method, device and medium based on deep learning model

Abstract

The present disclosure provides a data processing method, device, and medium based on a deep learning model, which relates to the field of artificial intelligence technology, and in particular to computing fields such as deep learning and large models. A specific implementation scheme is as follows: the deep learning model includes an expert network layer, and the expert network layer includes multiple expert networks; the processor divides the input features representing a word into multiple sub-features corresponding to the multiple expert networks respectively; based on the evaluation values ​​of the multiple sub-features, the processor determines at least one target sub-feature from the multiple sub-features; wherein the evaluation value represents the importance of the sub-feature relative to the input feature; based on the at least one target sub-feature and the network parameters of at least one target expert network corresponding to the at least one target sub-feature in the multiple expert networks, the processor performs a model training task or a model inference task to obtain a processing result.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to computing fields such as deep learning and large models. More specifically, the present disclosure provides a data processing method, electronic device, storage medium, and computer program product based on a deep learning model. Background Art

[0002] The development of the natural language field is moving towards the era of ultra-large-scale models. By using computing power to train ultra-large parameter models on massive text data, the resulting large models can have general semantic understanding and generation capabilities for multi-task and few-sample learning.

[0003] However, in the actual training and inference process, the extremely large number of parameters of large models leads to huge computational and storage overhead. Summary of the Invention

[0004] The present disclosure provides a data processing method, electronic device, storage medium, and computer program product based on a deep learning model.

[0005] According to one aspect of the present disclosure, a data processing method based on a deep learning model is provided, wherein the deep learning model includes an expert network layer, and the expert network layer includes multiple expert networks; a processor divides an input feature representing a word into multiple sub-features corresponding to the multiple expert networks respectively; based on the evaluation values ​​of the multiple sub-features, the processor determines at least one target sub-feature from the multiple sub-features; wherein the evaluation value represents the importance of the sub-feature relative to the input feature; based on the at least one target sub-feature and the network parameters of at least one target expert network corresponding to the at least one target sub-feature in the multiple expert networks, the processor executes a model training task or a model inference task to obtain a processing result.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided in the present disclosure when executed by a processor.

[0009] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0011] Figure 1 Schematic diagram of an application scenario of a data processing method and apparatus based on a deep learning model according to an embodiment of the present disclosure;

[0012] Figure 2 is a schematic flow chart of a data processing method based on a deep learning model according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic structural diagram of a deep learning model according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of a feedforward neural network submodel in a deep learning model according to an embodiment of the present disclosure;

[0015] Figure 5A~Figure 5B is a schematic diagram of an attention sub-model in a deep learning model according to an embodiment of the present disclosure;

[0016] Figure 6 It is a structural block diagram of an electronic device used to implement the data processing method based on the deep learning model of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0019] In the technical solution disclosed herein, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0020] In some technical solutions, a sparse Mixture of Experts (MoE) network can be used for the Feedforward Network (FFN) layer in the large model network structure. For example, the FFN's linear layer parameter matrix is ​​divided into multiple groups, each corresponding to an expert network. A routing network (Router) is also designed. During FFN data processing, the input word is called a token. The routing network calculates the correlation between the token and each expert network, and then routes the token to the expert network with high correlation for processing. At the same time, the token is not input to the expert network with low correlation for processing. In this way, by activating only some expert networks, the model parameter scale is expanded while keeping the activation parameters unchanged, thereby accelerating model training convergence and reducing inference costs.

[0021] However, this technical solution only targets FFNs and doesn't modify other networks, resulting in a limited scope of applicability. Furthermore, when processing data, highly correlated expert networks require processing the entire token. For example, if a token is a 4096-dimensional vector, the expert network needs to perform calculations on that 4096-dimensional vector. However, tokens have data redundancy. For example, some elements in the vector representing the token may be highly important and have a greater impact on the output, while others may be less important and have a smaller impact on the output, resulting in low computational efficiency.

[0022] The disclosed embodiments provide a data processing method based on deep learning models. Compared to the MoE method, which can only be modified for FFN structures, this method has a wider range of applications. Furthermore, this method can reduce processor computational efficiency, thereby accelerating model training and inference efficiency, and expanding the overall number of model parameters.

[0023] The technical solutions provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Figure 1 It is a schematic diagram of an application scenario of a data processing method and device based on a deep learning model according to an embodiment of the present disclosure.

[0025] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure may not be used in other devices, systems, environments or scenarios.

[0026] like Figure 1As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0027] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers, etc.

[0028] Server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., processing results obtained or generated based on user requests) to the terminal device.

[0029] It should be noted that the data processing method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the data processing apparatus provided in the embodiments of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the data processing apparatus provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0030] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0031] Figure 2 It is a schematic flowchart of a data processing method based on a deep learning model according to an embodiment of the present disclosure.

[0032] like Figure 2 As shown, the data processing method 200 is implemented based on a model, which can be a deep learning model, a large model, etc. Taking the deep learning model as an example, the model includes an expert network layer, and the expert network layer includes multiple expert networks. The data processing method 200 can include operations S210 to S230.

[0033] In operation S210 , the processor divides an input feature representing a word into a plurality of sub-features corresponding to a plurality of expert networks respectively.

[0034] The data to be processed can be text, audio, images, and more. During data processing, it must first be converted into a vector representation. For example, if the data to be processed is text, each character can be mapped to a token after conversion. For example, the text "data processing" can be converted into four tokens, each of which can be called an input feature.

[0035] The processor can split the input feature into multiple sub-features. For example, if an input feature is a 4096-dimensional row vector, the input feature can be split according to a predetermined length to obtain multiple sub-features. The dimensions of the multiple sub-features can be the same or different.

[0036] Multiple sub-features correspond to multiple expert networks. This correspondence can be one-to-one, one sub-feature can correspond to multiple expert networks, or one expert network can correspond to multiple sub-features. This embodiment does not limit the correspondence between sub-features and expert networks. The correspondence can be pre-configured. For example, the multiple sub-features after segmentation can form a sequence, where the first sub-feature in the sequence corresponds to the first expert network, and the second sub-feature in the sequence corresponds to the second expert network.

[0037] In operation S220 , the processor determines at least one target sub-feature from the plurality of sub-features based on respective evaluation values ​​of the plurality of sub-features, wherein the evaluation value represents the importance of the sub-feature relative to the input feature.

[0038] For example, the deep learning model may also include a routing network layer, which can be implemented based on a linear layer. The routing network layer can determine the evaluation value of each sub-feature based on the difficulty and information content of the input feature, and can determine the sub-feature with the higher evaluation value as the target sub-feature. The routing network layer is pre-trained, and this embodiment does not limit the training process.

[0039] For example, the evaluation value represents the importance of a sub-feature relative to the input features. Importance can indicate the impact of a sub-feature on the output of a deep learning model. A high evaluation value indicates that the sub-feature is relatively important and has a greater impact on the processing results. The absence of this sub-feature will significantly change the output of the deep learning model. A low evaluation value indicates that the sub-feature is less important and has a smaller impact on the processing results. Even if this sub-feature is missing, the output of the deep learning model will not change significantly. It can be seen that compared to sub-features with higher evaluation values, these sub-features with lower evaluation values ​​are relatively redundant and can therefore be left unprocessed.

[0040] In operation S230, the processor performs a model training task or a model inference task based on the at least one target sub-feature and a network parameter of at least one target expert network corresponding to the at least one target sub-feature among the multiple expert networks to obtain a processing result.

[0041] For example, the processor may operate on the network parameters of the target sub-feature target expert network having a corresponding relationship to obtain an operation result, and then determine the processing result based on the operation result of each target sub-feature.

[0042] According to the data processing based on the deep learning model provided by the embodiment of the present disclosure, during the data processing process, the processor divides the input feature into multiple sub-features, and then evaluates the multiple sub-features to obtain the evaluation value of each sub-feature. It should be noted that the correlation between the sub-features and the expert network is predetermined, rather than dynamically determined by the evaluation value. The evaluation value does not represent the correlation between the sub-features and the expert network, but the importance of the sub-feature. A high evaluation value indicates that the sub-feature is more important and has a greater impact on the output result of the deep learning model. A low evaluation value indicates that the sub-feature is less important and has a smaller impact on the output result of the deep learning model. The sub-feature with high importance is then determined as the target sub-feature. Next, based on the correspondence between the sub-features and the expert networks, a target expert network for processing the target sub-feature can be determined from multiple expert networks. The processing result can then be determined based on the network parameters of the target expert network, the target sub-features, and the correspondence between the target expert network and the target sub-features.

[0043] On the one hand, this embodiment is based on the Mixture of Hidden Dimension (MoH) and utilizes the sparsity of the hidden dimension to achieve conditional activation. It can be applied to any network result, such as FFN, attention mechanism, etc. Compared with the method in which MoE can only be modified for FFN structure, this embodiment has a wider scope of application.

[0044] On the other hand, this embodiment utilizes the redundant characteristics of the input features and the sparse characteristics of the hidden layer dimensions in the model structure. The target expert network processes not the complete input features, but the sub-features with high importance. At the same time, the sub-features with low importance are no longer involved in the calculation, thereby sparsely activating the hidden layer dimensions. In this way, while the number of overall model parameters remains unchanged, the amount of processor calculations can be reduced, the computing efficiency can be improved, and the training and reasoning efficiency of the model can be accelerated. While the activation parameters remain unchanged, the overall parameter amount of the model can be efficiently expanded, and the expansion of the overall parameter amount of the model complies with the characteristics of the Scaling Law, that is, the larger the overall parameter amount, the better the large model effect.

[0045] Figure 3 It is a schematic structural diagram of a deep learning model according to an embodiment of the present disclosure.

[0046] In this embodiment, the deep learning model Model may include at least one of an encoder and a decoder, the encoder or decoder may include a sub-model, the sub-model may include at least one expert network layer, and the expert network layer may include multiple expert networks. It should be noted that in actual application, the deep learning model Model can be implemented based on architectures such as Transformer and Bert, and this embodiment does not limit the results of the deep learning model Model. In addition, this embodiment does not limit the number of encoders, decoders, sub-models, expert network layers, and expert networks contained in the deep learning model Model.

[0047] like Figure 3 As shown, in one example, the deep learning model Model is a Transformer-based model, which may include an encoder and a decoder. Either the encoder or the decoder may include a sub-model, and the sub-model may include, for example, a feed-forward neural network sub-model FFN. The feed-forward neural network sub-model FFN may include at least one expert network layer. For example, the feed-forward neural network sub-model FFN may include an upsampling network layer, a downsampling network layer, and a gating network layer. Any one of the three network layers may be an expert network layer including multiple expert networks. It can be seen that this embodiment improves the feed-forward neural network sub-model FFN in the Transformer so that the feed-forward neural network sub-model FFN includes at least one expert network layer, and the expert network layer includes multiple expert networks. During the model training and inference process, only some expert networks may be activated, thereby reducing the computational complexity of the feed-forward neural network sub-model FFN.

[0048] like Figure 3As shown, in another example, the deep learning model Model is a Transformer-based model, which may include an encoder and a decoder. Either of the encoder and the decoder may include a sub-model, such as an attention sub-model Attention. The attention sub-model Attention may include at least one expert network layer. For example, the attention sub-model Attention may include a query network layer, a key network layer, a value network layer, and may also include a downsampling network layer. Any one of the four network layers may be an expert network layer including multiple expert networks. It can be seen that this embodiment improves the attention sub-model Attention in the Transformer so that the attention sub-model Attention includes at least one expert network layer, and the expert network layer includes multiple expert networks. During the model training and inference process, only some expert networks may be activated, thereby reducing the computational complexity of the attention sub-model Attention.

[0049] Figure 4 It is a schematic diagram of the feedforward neural network sub-model in the deep learning model according to an embodiment of the present disclosure.

[0050] like Figure 4 As shown, this embodiment takes the feedforward neural network sub-model FFN in Transformer as an example to illustrate the data processing method.

[0051] In this embodiment, the Transformer model includes an encoder and a decoder. Either the encoder or the decoder may include a sub-model. In this embodiment, the sub-model is a feed-forward neural network sub-model FFN.

[0052] According to another embodiment of the present disclosure, the feedforward neural network sub-model FFN may include a routing network layer Router1, and the routing network layer Router1 may be implemented based on a linear layer. The trained routing network layer Router1 is used to determine the evaluation value of each sub-feature in the input feature. For example, the input features of the feedforward neural network sub-model FFN may include multiple input features, and the multiple input features may include, for example, input features F1, F2, and F3. In the process of processing the input feature F2, the processor first divides the input feature F2 into N sub-features, for example, to obtain sub-features F20, F21, F22, F23, and F24. The routing network layer Router1 may be used to evaluate the sub-features F20, F21, F22, F23, and F24, thereby obtaining the evaluation values ​​of the sub-features F20, F21, F22, F23, and F24.

[0053] According to another embodiment of the present disclosure, the feedforward neural network sub-model FFN further includes an upsampling network layer L_up, which can be an expert network layer including multiple expert networks. Next, the working principle of the upsampling network layer L_up in the feedforward neural network sub-model FFN is described.

[0054] In this embodiment, the upsampling network layer L_up includes N expert networks, the N expert networks include M shared expert networks D_share1 and NM remaining expert networks D_up1, D_up2, D_up3, and D_up4, where N is an integer greater than or equal to 2, M is an integer greater than or equal to 1, and N>M. Figure 4 In the example, N is equal to 5 and M is equal to 1. It can be understood that in other embodiments, N and M can take other values.

[0055] The N expert networks correspond one-to-one to the N sub-features F20, F21, F22, F23, and F24 in the input feature F2. For example, sub-feature F20 corresponds to the shared expert network D_share1, sub-feature F21 corresponds to the expert network D_up1, sub-feature F22 corresponds to the expert network D_up2, sub-feature F23 corresponds to the expert network D_up3, and sub-feature F24 corresponds to the expert network D_up4.

[0056] In the process of determining the target sub-feature, sub-feature F20 corresponding to the shared expert network D_share1 can be used as the target sub-feature. In addition, among the sub-features F21, F22, F23, and F24 corresponding to the remaining expert networks D_up1, D_up2, D_up3, and D_up4, the sub-features whose evaluation values ​​meet the predetermined conditions can be determined as target sub-features, for example, sub-features F22 and F24 can be determined as target sub-features.

[0057] After obtaining the target sub-feature, the processor can perform a matrix operation on each target sub-feature of at least one target sub-feature based on the corresponding target sub-feature and the network parameters of the target expert network to obtain a first operation result, and then determine the processing result based on the first operation result. For example, a matrix multiplication operation is performed on the network parameters of the shared expert network D_share1 and the sub-feature F20, a matrix multiplication operation is performed on the network parameters of the expert network D_up2 and the sub-feature F22, and a matrix multiplication operation is performed on the network parameters of the expert network D_up4 and the sub-feature F24. In this way, three first operation results can be obtained. The three first operation results are the output information of the upsampling network layer L_up.

[0058] It can be seen that in this embodiment, sub-features F20, F22, and F24 are target sub-features, so the shared expert network D_share1 and the expert networks D_up2 and D_up4 are activated, and the target sub-features F20, F22, and F24 and the activated expert networks D_share1, D_up2, and D_up4 participate in the calculation. However, sub-features F21 and F23 are not target sub-features, so the expert networks D_up1 and D_up3 are not activated, and these sub-features F21, F23 and the inactivated expert networks D_up1 and D_up3 do not participate in the calculation. Compared to the case where all sub-features and all expert networks participate in the calculation, this embodiment can reduce the amount of calculation of the upsampling network layer L_up in the feedforward neural network sub-model FFN, and improve the computing efficiency of the processor.

[0059] It should be noted that Figure 4 In the example provided in, the upsampling network layer L_up includes a shared expert network D_share1, and the shared expert network D_share1 is not controlled by the routing network layer Router1. The shared expert network D_share1 is activated for each calculation, so that the shared information of each input feature is summarized on the shared expert network D_share1, ensuring the stability and robustness of the training and further improving the model effect. In other embodiments, the shared expert network D_share1 can be omitted. In this case, the input feature F2 is split to obtain multiple sub-features, and the multiple sub-features no longer contain the sub-features corresponding to the shared expert network D_share1. The sub-features whose evaluation values ​​meet the predetermined conditions among the multiple sub-features can be determined as target sub-features.

[0060] In some embodiments, the predetermined condition includes: the evaluation value of the sub-feature is greater than or equal to an evaluation value threshold. In other embodiments, the evaluation values ​​of multiple sub-features can be sorted based on the size of the evaluation values ​​to obtain an evaluation value sequence. If the order of the evaluation value of a sub-feature in the evaluation value sequence meets the order sub-condition, the sub-feature can be determined as a target sub-feature, where the order sub-condition may include: the order does not exceed a predetermined order K, and the predetermined order K may be 2. In this way, the two sub-features with the largest evaluation values ​​are determined as target sub-features. In this embodiment, the sub-feature with the higher evaluation value can be determined as the target sub-feature through the evaluation value threshold or order.

[0061] The above describes the working principle of the upsampling network layer L_up in the feedforward neural network sub-model FFN.

[0062] According to another embodiment of the present disclosure, the feedforward neural network sub-model FFN further includes a gated network layer L_gate, which can be an expert network layer including multiple expert networks. Next, the working principle of the gated network layer L_gate in the feedforward neural network sub-model FFN is described.

[0063] In this embodiment, the gated network layer L_gate includes N expert networks, including M shared expert networks and NM remaining expert networks, where N is an integer greater than or equal to 2, M is an integer greater than or equal to 1, and N>M. The N expert networks correspond one-to-one to the N sub-features. In addition, in the same feedforward neural network sub-model FFN, the target experts activated in different expert network layers correspond one-to-one. For example, in the upsampling network layer L_up, the shared expert network D_share1 and the expert networks D_up2 and D_up4 are activated. Then, in the gated network layer L_gate, the shared expert network corresponding to sub-feature F20, the expert network corresponding to sub-feature F22, and the expert network corresponding to sub-feature F24 are activated, while the expert network corresponding to sub-feature F21 and the expert network corresponding to sub-feature F23 are not activated. The working principle of the gated network layer L_gate is similar to that of the upsampling network layer L_up and will not be repeated in this embodiment.

[0064] The above describes the working principle of the gated network layer L_gate in the feedforward neural network sub-model FFN.

[0065] According to another embodiment of the present disclosure, the feedforward neural network sub-model FFN further includes a downsampling network layer L_down, which can be an expert network layer including multiple expert networks. Next, the working principle of the downsampling network layer L_down in the feedforward neural network sub-model FFN is described.

[0066] First, the input information of the downsampling network layer L_down is described. For example, the input feature F2 is processed by the upsampling network layer L_up and becomes the first output information of the upsampling network layer L_up. At the same time, the input feature F2 is processed by the gated network layer L_gate and becomes the second output information of the gated network layer L_gate. The second output information is processed by the activation function and then matrix-operated with the first output information. The result of the operation serves as the input of the downsampling network layer L_down.

[0067] For example, if three target expert networks are activated in the upsampling network layer L_up and the gated network layer L_gate, respectively, the upsampling network layer L_up can output three first operation results, and the gated network layer L_gate can output three operation results. After the activation function and matrix operation, three intermediate sub-features are obtained. The three intermediate sub-features are the input information of the downsampling network layer L_down. For the downsampling network layer L_down, the above-mentioned process of converting the target sub-features into intermediate sub-features using the upsampling network layer L_up, the gated network layer L_gate and the activation function can be called preprocessing. In other embodiments, preprocessing can also be achieved by other means, and this embodiment does not limit the preprocessing process.

[0068] The downsampling network layer L_down includes N expert networks, including M shared expert networks D_share2 and NM remaining expert networks D_down1, D_down2, D_down3, and D_down4, where N is an integer greater than or equal to 2, M is an integer greater than or equal to 1, and N>M. Figure 4 In the example, N is equal to 5 and M is equal to 1. It can be understood that in other embodiments, N and M can take other values.

[0069] Similar to the upsampling network layer L_up, the multiple expert networks in the downsampling network layer L_down correspond one-to-one to the multiple sub-features in the input features. For example, sub-feature F20 corresponds to the shared expert network D_share2, sub-feature F21 corresponds to the expert network D_down1, sub-feature F22 corresponds to the expert network D_down2, sub-feature F23 corresponds to the expert network D_down3, and sub-feature F24 corresponds to the expert network D_down4. In addition, the activated target expert networks in different expert network layers in the same sub-model correspond one-to-one. For example, if the shared expert network D_share2 and the expert networks D_up2 and D_up4 in the upsampling network layer L_up are activated, then in the downsampling network layer L_down, the shared expert network D_share2 and the expert networks D_down2 and D_down4 are also activated, while the expert networks D_down1 and D_down3 are not activated.

[0070] After obtaining the intermediate sub-features, the processor can perform a matrix operation based on the intermediate sub-features and the network parameters of the target expert network corresponding to the target sub-features to obtain a second operation result. For example, pre-processing the target sub-features F20, F22, and F24 can obtain the first intermediate sub-feature, the second intermediate sub-feature, and the third intermediate sub-feature. The first intermediate sub-feature can be matrix-operated with the network parameters of the shared expert network D_share2 to obtain the first second operation result, the second intermediate sub-feature can be matrix-operated with the network parameters of the expert network D_down2 to obtain the second second operation result, and the third intermediate sub-feature can be matrix-operated with the network parameters of the expert network D_down4 to obtain three second operation results. The three second operation results are the output information of the downsampling network layer L_down.

[0071] The above describes the working principle of the downsampling network layer L_down in the feedforward neural network sub-model FFN.

[0072] According to another embodiment of the present disclosure, the feedforward neural network sub-model FFN further includes a merging network layer Merge1. Next, the working principle of the merging network layer Merge1 in the feedforward neural network sub-model FFN is explained.

[0073] In one example, multiple second operation results output by the downsampling network layer L_down can be directly fused to obtain fused data, and the fusion method can be splicing. Then, based on the merging network layer Merge1, the processor processes the fused data into output information of the merging network layer Merge1. In addition, the dimension of the output information of the merging network layer Merge1 can be the same as the dimension of the input feature.

[0074] In another example, the value of at least one second operation result may be adjusted based on the target weight to obtain the adjusted at least one second operation result. The processor then fuses the adjusted at least one second operation result to obtain fused data, where the fusion method may be splicing. Based on the merged network layer Merge1, the processor then processes the fused data into output information of the merged network layer Merge1.

[0075] It should be noted that, compared with the case where the input features are not split, the splitting of the input features in this embodiment is likely to cause the numerical value of the output information of the downsampling network layer L_down to be reduced. Therefore, the predetermined scaling parameter can be used as the target weight to expand the numerical value of the output information of the downsampling network layer L_down. Alternatively, the evaluation value output by the router can be used as the target weight so that the adjusted second operation result reflects the importance of the target sub-token. Alternatively, the product, weighted sum, or other calculation results of the predetermined scaling parameter and the evaluation value output by the router can be used as the target weight. Adjustment can be made based on a multiplication operation, for example, by multiplying the target weight by the second operation result to obtain an adjusted second operation result.

[0076] After obtaining the output information of the merged network layer Merge1 in the above manner, the output information of the merged network layer Merge1 can be used as the output information of the sub-model. In this embodiment, the merged network layer Merge1 can use a linear mapping method to map the input to the output information, and map its own output information to the original dimension of the input feature. In practical applications, for the Transformer model, the Transformer model can include multiple cascaded sub-models, which can ensure that the latter sub-model can smoothly process the output information of the previous sub-model.

[0077] The above describes the working principle of the merge network layer Merge1 in the feedforward neural network sub-model FFN.

[0078] In some embodiments, for the feedforward neural network sub-model, the processing results for the input features may include at least one of the output information of the upsampling network layer, the output information of the gated network layer, the output information of the downsampling network layer, the output information of the merged network layer, and the like.

[0079] Figure 5A~Figure 5B It is a schematic diagram of the attention sub-model in the deep learning model according to an embodiment of the present disclosure.

[0080] like Figure 5A~Figure 5B As shown in FIG, this embodiment takes the attention sub-model Attention in Transformer as an example to illustrate the data processing method.

[0081] In this embodiment, the Transformer model includes an encoder and a decoder. Either the encoder or the decoder may include a sub-model. In this embodiment, the sub-model is an attention sub-model.

[0082] According to another embodiment of the present disclosure, the attention sub-model Attention may include a routing network layer Router2. The structure and working principle of the routing network layer Router2 may refer to the routing network layer Router1 in the feedforward neural network sub-model FFN, and will not be repeated in this embodiment.

[0083] According to another embodiment of the present disclosure, the attention sub-model Attention may include a query network layer Q, which may be an expert network layer including multiple expert networks. For example, the query network layer Q includes a shared expert network D_share3 and NM remaining expert networks D_Q1, D_Q2, D_Q3, and D_Q4. The working principle of the query network layer Q is similar to that of the upsampling network layer L_up in the feedforward neural network sub-model FFN mentioned above. During data processing, multiple input features F4, F5, and F6 can be obtained. Taking the input feature F5 as an example, the input feature F5 is divided into multiple sub-features F50, F51, F52, F53, and F54. The expert network in the query network layer Q corresponds one-to-one to the multiple sub-features. For example, the sub-features F50, F52, and F54 among the multiple sub-features are determined as target sub-features. At this time, the expert networks D_share3, D_Q2, and D_Q4 are activated as target experts. The output information of the query network layer Q can be obtained through matrix operations.

[0084] According to another embodiment of the present disclosure, the attention sub-model Attention may include a key network layer K, and the key network layer K may be an expert network layer including multiple expert networks. For example, the key network layer K includes a shared expert network D_share4 and NM remaining expert networks D_K1, D_K2, D_K3, and D_K4. The working principle of the key network layer K is similar to that of the query network layer Q mentioned above. During the data processing, the expert networks D_share4, D_K1, D_K2, D_K3, and D_K4 in the key network layer K correspond one-to-one to multiple sub-features F50, F51, F52, F53, and F54. The expert networks D_share4, D_K2, and D_K4 can be used as activated target experts. The output information of the key network layer K can be obtained through matrix operations.

[0085] According to another embodiment of the present disclosure, the attention sub-model Attention may include a value network layer V, which may be an expert network layer including multiple expert networks. For example, the value network layer V includes a shared expert network D_share5 and NM remaining expert networks D_V1, D_V2, D_V3, and D_V4. The working principle of the value network layer V is similar to that of the query network layer Q mentioned above. During data processing, the expert networks D_share5, D_V1, D_V2, D_V3, and D_V4 in the value network layer V correspond one-to-one to multiple sub-features F50, F51, F52, F53, and F54. The expert networks D_share5, D_V2, and D_V4 can be used as the activated target expert networks. The output information of the value network layer V can be obtained through matrix operations.

[0086] According to another embodiment of the present disclosure, the attention sub-model Attention may include a downsampling network layer L_O, which may be an expert network layer including multiple expert networks. The output information of the query network layer Q, the output information of the key network layer K, and the output information of the value network layer V are processed based on the attention mechanism and can be input as output information to the downsampling network layer L_O. The downsampling network layer L_O includes a shared expert network D_share6 and NM remaining expert networks D_O1, D_O2, D_O3, and D_O4. The working principle of the value network layer V is similar to that of the downsampling network layer L_down in the feedforward neural network sub-model FFN mentioned above. During the data processing process, the expert networks D_share6, D_O1, D_O2, D_O3, and D_O4 in the value network layer V correspond one-to-one to multiple sub-features F50, F51, F52, F53, and F54. The expert networks D_share6, D_O2, and D_O4 can be used as activated target experts. The sub-features are processed by the attention mechanism and then enter the downsampling network layer L_O as input. For the downsampling network layer L_O, the processing of the sub-features before entering the downsampling network layer L_O can be called preprocessing. Then, matrix operations are performed on the input and network parameters of the downsampling network layer L_O to obtain the output information of the downsampling network layer L_O.

[0087] According to another embodiment of the present disclosure, the attention sub-model Attention may include a merging network layer Merge2, which may map its own output information to the original dimension of the input features. The working principle of the merging network layer Merge2 is similar to that of the merging network layer Merge1 in the feedforward neural network sub-model FFN described above, and will not be repeated in this embodiment. The output information of the merging network layer Merge2 may be used as the output information of the attention sub-model Attention.

[0088] In some embodiments, for the attention sub-model, the processing results for the input features may include at least one of the output information of the query network layer, the output information of the key network layer, the output information of the value network layer, the output information of the downsampling network layer, the output information of the merged network layer, and the like.

[0089] According to another embodiment of the present disclosure, network parameters of multiple expert networks are obtained by dividing network parameters of the expert network layer based on rows.

[0090] It should be noted that the dimension of the original parameter matrix of the upsampling network layer in FFN is D*D', and the dimension of the original parameter matrix of the downsampling network layer is D'*D.

[0091] In some technical solutions, MoE usually divides the original parameter matrix of the upsampling network layer in FFN into multiple groups by column, and divides the original parameter matrix of the downsampling network layer into multiple groups by row, each group is an expert network. This division method will reduce the output feature dimension of the expert network layer. For example, if the dimension of the token is 1*D and the original parameter matrix is ​​D*D', if it is divided into 4 expert networks, the dimension of the network parameter matrix of each expert network is D*1 / 4D'. The multiplication operation of the vector and the matrix is ​​to multiply the rows of the vector and the columns of the matrix. In this way, the data dimension obtained after the operation of the token and the network parameter matrix of the activated expert network is 1*1 / 4D'. Taking the example of two activated expert networks, the two dimensions are both 1*1 / 4D'. Then, the two data can be added as the output of the expert network layer. It can be seen that the data dimension after the addition operation is still 1*1 / 4D'.

[0092] In contrast, in this embodiment, in FFN, the network parameter matrix of the upsampling network layer can be split by rows. This can increase the dimension of the output features compared to the MoE method of splitting the original parameter matrix by columns. For example, if the dimension of the token is 1*D and the original parameter matrix of the upsampling network layer is D*D', if it is divided into 4 expert networks, then after splitting the token, 4 sub-features are obtained, and the dimension of each sub-feature is 1*1 / 4D. At the same time, in the upsampling network layer of FFN, the dimension of the network parameter matrix of each expert network is 1 / 4D*D', and the dimension of the data obtained after the token and the network parameter matrix of the activated expert network are calculated is 1*D'. It can be seen that the dimension of the output data obtained by the MoE method of splitting the original parameter matrix by columns is 1*1 / 4D', while the dimension of the output data obtained by the method of splitting the original parameter matrix by columns in this embodiment is 1*D'. The data dimension is larger, so from the perspective of data dimension, the information contained in the data is more comprehensive. The network parameter matrix of the downsampling network layer is split by columns. In the downsampling network layer, the dimension of the expert network is D'*1 / 4D, so that the input information of the downsampling network layer and the network parameters of the expert network can be directly operated on by matrices.

[0093] Similarly, in the attention sub-model, the query network layer, key network layer, and value network layer can split the original network parameter matrix by row and the network parameter matrix of the downsampling network layer by column. The working principle is similar to that of FFN and will not be repeated in this embodiment.

[0094] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned data processing method.

[0095] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above-mentioned data processing method.

[0096] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, which implements the above data processing method when executed by a processor.

[0097] Figure 6: This is a block diagram of an electronic device for implementing the data processing method based on a deep learning model of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0098] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. Computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0099] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0100] The computing unit 601 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the data processing method by any other suitable means (e.g., via firmware).

[0101] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0105] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0106] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0107] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0108] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data processing method based on a deep learning model, wherein the deep learning model includes an expert network layer, and the expert network layer includes multiple expert networks; The processor divides an input feature representing a word into a plurality of sub-features corresponding to the plurality of expert networks respectively; the input feature is obtained by performing data conversion on the data to be processed, and the data to be processed is text, audio, or image; The processor determines at least one target sub-feature from the plurality of sub-features according to respective evaluation values ​​of the plurality of sub-features; wherein the evaluation value represents the importance of the sub-feature relative to the input feature; According to the at least one target sub-feature and the network parameters of at least one target expert network corresponding to the at least one target sub-feature in the multiple expert networks, the processor performs a model training task or a model inference task to obtain a processing result.

2. The method according to claim 1, wherein The processor performs a model training task or a model inference task based on the at least one target sub-feature and a network parameter of at least one target expert network corresponding to the at least one target sub-feature in the multiple expert networks, and obtains a processing result including: For each target sub-feature of the at least one target sub-feature, the processor performs a matrix operation according to the corresponding target sub-feature and the network parameters of the target expert network to obtain a first operation result; The processing result is determined according to the first operation result.

3. The method according to claim 2, wherein: Determining the processing result according to the first operation result includes: The processor determines the first operation result as the output information of the expert network layer; and the processing result includes the output information of the expert network layer.

4. The method according to claim 1, wherein The processor performs a model training task or a model inference task based on the at least one target sub-feature and a network parameter of at least one target expert network corresponding to the at least one target sub-feature in the multiple expert networks, and obtains a processing result including: For each target sub-feature of the at least one target sub-feature, the processor pre-processes the target sub-feature to obtain an intermediate sub-feature; The processor performs a matrix operation based on the intermediate sub-features and the network parameters of the target expert network corresponding to the target sub-features to obtain a second operation result; The processing result is determined according to the second operation result.

5. The method according to claim 4, wherein The deep learning model includes a sub-model, and the sub-model includes the expert network layer and the merged network layer; and determining the processing result according to the second operation result includes: The processor fuses at least one second operation result determined based on the at least one target sub-feature to obtain fused data; Based on the merged network layer, the processor processes the fused data into output information of the sub-model; wherein the processing result includes the output information of the sub-model, and the dimension of the output information of the sub-model is the same as the dimension of the input feature.

6. The method according to claim 5, wherein: The processor fuses at least one second operation result determined based on the at least one target sub-feature to obtain fused data, including: The processor adjusts the value of the at least one second operation result based on the target weight to obtain the at least one adjusted second operation result; The processor fuses the adjusted at least one second operation result to obtain the fused data.

7. The method according to claim 6, wherein: The target weight is determined based on at least one of the evaluation value of the target sub-feature and a predetermined scaling parameter.

8. The method according to claim 1, wherein The multiple expert networks include a shared expert network and remaining expert networks other than the shared expert network; the multiple sub-features include: a first sub-feature corresponding to the shared expert network and a second sub-feature corresponding to the remaining expert network; the processor determining at least one target sub-feature from the multiple sub-features based on respective evaluation values ​​of the multiple sub-features includes: The processor determines the first sub-feature as the target sub-feature; Among the second sub-features, the processor determines the sub-feature whose evaluation value meets a predetermined condition as the target sub-feature.

9. The method according to claim 1, wherein: The processor determining at least one target sub-feature from the plurality of sub-features according to respective evaluation values ​​of the plurality of sub-features comprises: The processor determines, among the multiple sub-features, a sub-feature whose evaluation value satisfies a predetermined condition as the target sub-feature.

10. The method according to any one of claims 8 to 9, wherein: The predetermined condition includes one of the following: The evaluation value of the sub-feature is greater than or equal to an evaluation value threshold; In the evaluation value sequence, the order of the evaluation values ​​of the sub-features satisfies an order sub-condition; wherein the evaluation value sequence is obtained by sorting the evaluation values ​​of the plurality of sub-features based on the magnitude of the evaluation values.

11. The method according to claim 1, wherein The processor divides the input feature representing a word into a plurality of sub-features corresponding to the plurality of expert networks respectively, including: The processor divides the input feature into the plurality of sub-features according to a predetermined length.

12. The method according to any one of claims 1 to 11, wherein: The deep learning model includes at least one sub-model, and the sub-model includes multiple expert network layers; in the same sub-model, the target expert networks in any two expert network layers correspond one to one.

13. The method according to claim 12, wherein: The deep learning model is a Transformer-based model, the at least one sub-model includes a feedforward neural network sub-model, and the multiple expert network layers in the feedforward neural network sub-model include: an upsampling network layer, a downsampling network layer and a gating network layer.

14. The method according to claim 12, wherein: The deep learning model is a Transformer-based model, the at least one sub-model includes an attention sub-model, and the multiple expert network layers in the attention sub-model include: a query network layer, a key network layer, a value network layer and a downsampling network layer.

15. The method according to claim 12, wherein: The sub-model also includes a routing network layer; the method further includes: Based on the routing network layer, the processor determines the evaluation value of each of the plurality of sub-features.

16. The method according to any one of claims 1 to 15, wherein: The network parameters of the multiple expert networks are obtained by dividing the network parameters of the expert network layer based on rows.

17. The method according to claim 1, wherein The multiple sub-features in the input features correspond one-to-one to the multiple expert networks in the expert network layer.

18. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 17.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 17.

20. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 17.

Images