Modal unified data processing method and device, equipment and storage medium

Abstract

The invention provides a modal unified data processing method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, large models and the like. The specific implementation scheme is as follows: determining a plurality of first weight parameters corresponding to a plurality of attention experts in an attention layer by using a first gating network according to vector features of input lexical elements; based on the plurality of first weight parameters, processing the input lexical element by using at least one activated attention expert in the plurality of attention experts to obtain an attention layer output vector; using the first residual connection and the normalization layer to generate an intermediate lexical element according to the attention layer output vector and the input lexical element; and processing the intermediate lexical elements by using the feed-forward network layer to generate output lexical elements. According to the scheme of the embodiment of the invention, intermodal interference can be effectively relieved, and a single model can flexibly and efficiently process multi-modal and multi-task data.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning and large models. Background Art

[0002] Large language models (LLMs) have demonstrated strong capabilities when processing data from a single modality, such as text. However, real-world scenarios are complex and diverse, and single text descriptions often fail to accurately capture them. Consequently, LLMs are unable to accurately respond. The simplest and most efficient way to address this issue is to equip the model with the ability to process input from other modalities. However, when moving towards multimodal development, existing models often face interference between different modalities and tasks. Summary of the Invention

[0003] The present disclosure provides a modality-unified data processing method, apparatus, device, and storage medium.

[0004] According to one aspect of the present disclosure, a modality-unified data processing method is provided, comprising:

[0005] Determining, using the first gating network, a plurality of first weight parameters corresponding to a plurality of attention experts in the attention layer according to the vector features of the input word unit;

[0006] Based on the plurality of first weight parameters, using at least one activated attention expert from the plurality of attention experts to process the input word to obtain an attention layer output vector;

[0007] Using the first residual connection and layer normalization layer, the intermediate word is generated according to the output vector of the attention layer and the input word;

[0008] The intermediate word-grams are processed using the feed-forward network layer to generate the output word-grams.

[0009] According to another aspect of the present disclosure, there is provided a device for modality-unified data processing, comprising:

[0010] an allocation module, configured to determine, using the first gating network, a plurality of first weight parameters corresponding to a plurality of attention experts in the attention layer according to the vector features of the input word unit;

[0011] an attention module, configured to process the input word-unit using at least one activated attention expert from the plurality of attention experts based on the plurality of first weight parameters to obtain an attention layer output vector;

[0012] A first generation module is configured to generate an intermediate word according to the output vector of the attention layer and the input word using a first residual connection and a layer normalization layer;

[0013] The second generation module is used to process the intermediate word units using the feedforward network layer to generate output word units.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0020] According to the solution of the embodiment of the present disclosure, inter-modal interference can be effectively alleviated, enabling a single model to flexibly and efficiently process multi-modal and multi-task data.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0023] Figure 1 1 is a flow chart of a modality-unified data processing method provided according to an embodiment of the present disclosure;

[0024] Figure 2 is a structural diagram of a converter module for modal unification provided according to an embodiment of the present disclosure;

[0025] Figure 3 Schematic diagram of the structure of the MoE-MHSA layer provided according to one embodiment of the present disclosure;

[0026] Figure 4 is a schematic structural diagram of a MoE-MHSA layer with shared experts provided according to an embodiment of the present disclosure;

[0027] Figure 5 is a schematic structural diagram of a MoE-FFN layer provided according to an embodiment of the present disclosure;

[0028] Figure 6 is a flowchart of a modality-unified data processing method provided according to another embodiment of the present disclosure;

[0029] Figure 7 is a schematic structural diagram of a MoE-FFN layer with shared experts provided according to an embodiment of the present disclosure;

[0030] Figure 8 This is a flow chart of a modality-unified data processing device according to an embodiment of the present disclosure.

[0031] Figure 9 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] In related technologies, for natural language, words are independent and the number of words is limited, so they can be more naturally modeled as word vector groups. However, for images, there is correlation between adjacent pixels, there is a lot of redundant information, and coupled with the continuous nature of images, it is difficult to model information losslessly like text. This difference in modality itself leads to a certain amount of interference when the model processes features of different modalities. This interference is more obvious in the two tasks of image understanding and image generation: image understanding focuses on text output, and image generation focuses on image output. If a single model is used to handle these two tasks, the performance of each task will be reduced. In addition, most existing mixture of experts (MoE) models are only modified for the feedforward neural network (FFN) layer, and their scope of application is limited.

[0034] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the embodiments of the present disclosure provide a modality-unified data processing method. By utilizing the technical solutions of the embodiments of the present disclosure, inter-modal interference can be effectively alleviated, enabling a single model to flexibly and efficiently process multi-modal and multi-task data.

[0035] Figure 1This is a flow chart of a modality-unified data processing method according to an embodiment of the present disclosure. This method can be used for data processing in a deep learning model, which can usually be implemented based on a Transformer architecture. Figure 1 As shown, the method comprises at least the following steps:

[0036] S110. Utilize the first gating network to determine, based on the vector features of the input word, a plurality of first weight parameters corresponding to a plurality of attention experts in the attention layer.

[0037] The present disclosure provides a method for processing data within a deep learning model, which can generally be implemented based on a Transformer architecture. The model includes an improved Transformer building block (Block), such as Figure 2 As shown in Figure 1, this module includes a Mixture of Experts Multi-Head Self-Attention (MoE-MHSA) layer, a first residual connection and layer normalization layer, a Mixture of Experts Feed-Forward Network (MoE-FFN) layer, and a second residual connection and layer normalization layer. This is hereinafter referred to as the Uni-Mod Transformer Block.

[0038] One or more Uni-Mod Transformer Blocks in the model process the input data. Specifically, the input to this module is a token, which is the basic processing unit formed by encoding input data (such as text, images, etc.) and exists in the form of a vector in the model.

[0039] In a typical processing flow, when an input word enters the module, it is first processed by an attention layer. Within the attention layer, a first gating network is implemented. This gating network is a small neural network that analyzes the vector features of the input word and outputs a set of first weight parameters. These weight parameters correspond to multiple attention experts pre-set in the attention layer. Each expert can be thought of as a subnetwork specialized in processing specific features. The function of the gating network is to determine, based on the content of the word itself, which experts should process it and the weight that each selected expert's results should receive.

[0040] S120. Based on multiple first weight parameters, use at least one activated attention expert from multiple attention experts to process the input word to obtain an attention layer output vector.

[0041] Subsequently, based on the first weight parameter output by the gating network, one or more attention experts with the highest scores are activated to process the input word. The results of these activated experts are fused to form an attention layer output vector. This process enables conditional and dynamic allocation of computing resources.

[0042] S130. Utilize the first residual connection and the layer normalization layer to generate an intermediate word based on the attention layer output vector and the input word.

[0043] Next, the attention layer output vector passes through the first residual connection and layer normalization layers. Residual connections are a shortcut that adds the module input directly to the output, effectively preventing performance degradation as the network depth increases. Layer normalization is a technique that stabilizes the training process. This step results in an intermediate word.

[0044] S140: Process the intermediate word-units using the feedforward network layer to generate output word-units.

[0045] Finally, this intermediate word, infused with contextual information, is fed into a feedforward network layer. This layer performs further feature extraction and nonlinear transformation on the intermediate word to generate the final output word. This output word can then be fed into the next identical processing module or used for the final task output.

[0046] According to the solution of the embodiment of the present disclosure, a gating network is used to dynamically assign experts to each word, allowing the model to flexibly process data based on its own characteristics, rather than using fixed, hard-coded rules. This greatly reduces interference between different modalities and tasks, and improves the performance of a single model in handling multiple tasks.

[0047] In one possible implementation, multiple attention experts each include an independent set of projection parameters for generating query, key, and value vectors to form an independent projection path.

[0048] In the disclosed embodiments, traditional attention mechanisms use a fixed set of projection matrices to transform input vectors into query, key, and value vectors. In this solution, the design of multiple attention experts means that multiple such pipelines are built into the model. Each pipeline has an independent set of projection parameters, thus forming an independent projection path.

[0049] In the attention mechanism, the model needs to generate three key vectors for each word: the query vector (Query, Q), the key vector (Key, K), and the value vector (Value, V). When the input word is assigned to an expert, the expert uses its unique set of projection parameters (which can be understood as three independent linear transformation layers) to map the input word vector into a set of corresponding Q, K, and V vectors.

[0050] like Figure 3 As shown in the figure, it is assumed that there are two attention experts in the attention layer, Expert 1 (QKV Proj1) and Expert 2 (QKVProj 2). Expert 1 has a set of QKV projection parameters, and Expert 2 has a completely different set of QKV projection parameters. This design allows different experts to learn to focus on and extract features at different levels. For example, one expert may be better at processing visual features related to texture, while another expert may be better at processing features related to contours or colors. When processing an image token containing a complex scene, the gating network can dynamically activate the most suitable combination of experts, thereby achieving more refined feature extraction.

[0051] The solutions in the embodiments of this disclosure allow the model to learn more decoupled and specialized feature transformation capabilities, enriching the model's expressive power and allowing it to select the most appropriate QKV generation method for tokens with different characteristics, thereby more precisely capturing complex associations in the data. For example, one expert might focus on learning positional relationships, while another might focus on learning semantic associations, thereby improving the overall representational capacity and flexibility of the model.

[0052] In one possible implementation, S120 processes the input word-unit using at least one activated attention expert from among the multiple attention experts based on the multiple first weight parameters to obtain an attention layer output vector, including:

[0053] S121. Based on multiple first weight parameters, use at least one activated attention expert among multiple attention experts to process the input word to obtain a first set of query key-value vectors.

[0054] S122. Based on the first set of query key-value vectors, perform multi-head self-attention calculation to obtain the attention layer output vector.

[0055] In the disclosed embodiment, the attention experts corresponding to all non-zero parameters in the multiple first weight parameters can be used as activated attention experts, or the K experts with the highest scores can be selected according to a preset strategy (such as Top-K routing, where K is usually 1 or 2). For example, in a typical setting, the two experts with the highest weights may be selected. Then, the input word will pass through the independent projection paths of the two activated experts respectively, generating two sets of QKV vectors. These two sets of vectors will be weighted and summed according to their weight parameters, and finally fused into the first set of query key-value vectors.

[0056] The first set of query key-value vectors is then fed into a standard multi-head self-attention module. In this module, the Q vector is similarity-computed with all K vectors, and the resulting similarity scores are used to perform a weighted sum of the V vectors. This process occurs in parallel across multiple heads, each focusing on different aspects of the information. Finally, the results from all heads are concatenated and linearly transformed to produce the final attention layer output vector.

[0057] According to the solution of the embodiment of the present disclosure, the model not only utilizes the flexibility of the MoE architecture to selectively extract features, but also retains the powerful contextual information integration capability of the multi-head self-attention mechanism.

[0058] In one possible implementation, S120 processes the input word-unit using at least one activated attention expert from among the multiple attention experts based on the multiple first weight parameters to obtain an attention layer output vector, including:

[0059] S123. Based on multiple first weight parameters, use at least one activated attention expert among the multiple attention experts to process the input word to obtain a first set of query key-value vectors.

[0060] S124. Use the first shared expert in the attention layer to process the input word to obtain a second set of query key-value vectors.

[0061] S125. Based on the first set of query key-value vectors and the second set of query key-value vectors, obtain the attention layer output vector.

[0062] In this disclosed embodiment, to further promote information fusion between different modalities or features, in addition to the multiple attention experts controlled by the aforementioned gating network, this solution also introduces a first shared expert in the attention layer. The introduction of the shared expert is intended to ensure that the model does not lose the basic information shared by all data while performing personalized and sparse learning.

[0063] Unlike regular experts that require activation by a gating network to work, the shared expert is activated by default for all input tokens. It does not select inputs but processes each token passing through the layer. Like other experts, it also has an independent set of QKV projection parameters.

[0064] like Figure 4 As shown in Figure 2, when an input token enters the attention layer, it is processed in parallel by two paths:

[0065] Path A (activate experts): Same as the description of S121 to S122, routed through the first gating network, processed by one or more activated experts, and obtains a first set of query key-value vectors.

[0066] Path B (Shared Expert): Meanwhile, all input tokens, regardless of whether they activate other experts, must be processed by this first shared expert. This shared expert, with its own independent projection parameters, produces a second set of query key vectors. Finally, the model jointly computes the final attention layer output vector based on these two sets of query key vectors.

[0067] According to the solution of the disclosed embodiment, all modal data passes through the shared expert, which can capture inter-modal correlations and promote information fusion between modalities. Furthermore, for certain modalities with less data, the shared expert can also draw information from the context of other modalities, enhancing their expressiveness. Finally, the shared expert can prevent the activatable experts from extracting homogeneous information, allowing them to focus on different features and tasks, thereby improving their utilization.

[0068] In one possible implementation, S125 obtains the attention layer output vector based on the first set of query key value vectors and the second set of query key value vectors, further including:

[0069] Aggregate the first set of query key value vectors and the second set of query key value vectors.

[0070] The aggregation results are subjected to multi-head self-attention calculation to obtain the attention layer output vector.

[0071] In the disclosed embodiment, the first set of query key-value vectors and the second set of query key-value vectors need to be aggregated. Aggregation is an information merging operation. For example, the two sets of vectors can be stacked or concatenated along the feature dimension to form a set of aggregated QKV vectors with a wider dimension and richer information. The aggregated Q, K, and V vectors can be viewed as enhanced vectors that contain both specialized features (from general experts) and universal features (from shared experts).

[0072] The model then uses this aggregated result as input to perform a multi-head self-attention calculation. The output of this calculation is the final attention layer output vector.

[0073] According to the solution of the embodiment of the present disclosure, shared experts ensure that the model can stably learn the basic features common to all data, while the activated experts focus on processing more personalized sparse features. The combination of the two can achieve better performance than a single mode.

[0074] In one possible implementation, S140 processes the intermediate word-grams using a feedforward network layer to generate output word-grams, including:

[0075] S141. Determine, using the second gating network and based on the vector features of the intermediate word, a plurality of second weight parameters corresponding to a plurality of feedforward network experts in the feedforward network layer.

[0076] S142. Based on the multiple second weight parameters, use at least one activated feedforward network expert among the multiple feedforward network experts to process the intermediate word to obtain a feedforward network expert output vector.

[0077] S143. Utilize the second residual connection and the layer normalization layer to generate output word units based on the feedforward network expert output vector and the intermediate word units.

[0078] In traditional Transformer modules, the FFN layer is typically a fixed network consisting of two linear layers. However, in the modality-unified Transformer module of the present disclosure, the FFN layer also adopts a mixture of experts (MoE) architecture, which can be called MoE FFN.

[0079] When the intermediate token is fed into the MoE FFN layer, the processing flow is very similar to the previous attention layer:

[0080] First, the intermediate word passes through a second gating network. This gating network determines a set of second weight parameters corresponding to multiple feedforward network experts (FFN Experts) in the FFN layer based on the vector features of the intermediate word. Each feedforward network expert is itself an independent feedforward network.

[0081] Then, according to the second weight parameter, one or more feedforward network experts are activated and process the intermediate word to obtain a feedforward network expert output vector.

[0082] Finally, this output vector is combined with the input of the layer (i.e., the intermediate word) through a second residual connection and a layer normalization layer to finally generate the output word of the entire modality unified Transformer module.

[0083] According to the solution of the embodiment of the present disclosure, the MoE structure is also applied to the FFN layer, further enhancing the capacity and flexibility of the model, enabling the model to perform dynamic and adaptive calculations in higher-dimensional feature spaces, thereby more effectively processing complex multimodal and multi-task data.

[0084] In one possible implementation, S142 processes the intermediate word-unit using at least one activated feedforward network expert from among the multiple feedforward network experts based on the multiple second weight parameters to obtain a feedforward network expert output vector, including:

[0085] An activated feedforward network expert is determined from a plurality of feedforward network experts according to the plurality of second weight parameters.

[0086] Intermediate tokens are processed using an activated feedforward network expert.

[0087] The processing results of the activated feedforward network experts are weighted and fused according to a plurality of second weight parameters to obtain a feedforward network expert output vector.

[0088] In the embodiment of the present disclosure, after the intermediate word passes through the second gating network and generates the second weight parameters, the model selects the expert based on these weight parameters. A common strategy is Top-k routing, where k is a preset integer (such as 1 or 2). Figure 5 As shown, feedforward network expert 1 (FFN 1) and feedforward network expert 2 (FFN 2) are included. Among the weight parameters output by the gating network, the feedforward network expert with the highest value is FFN 2, and FFN 2 will be determined as the activated feedforward network expert.

[0089] The intermediate word is then sent to FFN 2 for processing, and a result vector is output.

[0090] If multiple feedforward network experts are activated, to obtain a single "feedforward network expert output vector," the processing results of these activated experts must be weighted and fused. The fusion weights are the second weight parameters obtained by each of them from the second gating network. For example, the output vector of expert 1 is multiplied by its weight, and the output vector of expert 2 is multiplied by its weight, and then the two are added together to obtain the final feedforward network expert output vector.

[0091] According to the solution of the embodiment of the present disclosure, the fusion method based on weight parameters can better reflect the importance of different experts than simple averaging or summing, and is a smooth and efficient routing mechanism.

[0092] In one possible implementation, Figure 6 As shown, the data processing method further includes the steps of:

[0093] S610 , using a second shared expert in the feedforward network layer to process the intermediate word to obtain a shared expert output vector.

[0094] S143 uses the second residual connection and the layer normalization layer to generate output word units based on the feedforward network expert output vector and the intermediate word units, specifically including:

[0095] S620: Aggregate the feedforward network expert output vector and the shared expert output vector to obtain a second output vector.

[0096] S630: Generate an output word unit according to the second output vector and the intermediate word unit by using a second residual connection and a layer normalization layer.

[0097] In the embodiment of the present disclosure, the MoE feedforward network layer is further optimized by introducing a second shared expert. This design is similar to the shared expert logic in the attention layer. Figure 7 As shown, in this embodiment, when the intermediate word enters the feedforward network layer, it is processed by two paths at the same time: one path passes through the gating network and the activated expert (feedforward network expert 2 in the figure) to obtain the feedforward network expert output vector; the other path fixedly passes through the second shared expert to obtain the shared expert output vector.

[0098] These two output vectors are then aggregated. Aggregation can be a simple element-by-element addition operation, resulting in a second output vector. Finally, during the second residual connection and layer normalization, this aggregated second output vector is fused with the input intermediate token to generate the final output token.

[0099] According to the solution of the embodiment of the present disclosure, by introducing shared experts in the FFN layer, the model can also combine general knowledge and professional knowledge when performing high-order feature transformation, making the model more adaptable to different tasks and modalities, and further improving the representation ability.

[0100] In a possible implementation, the input word contains at least one modality information.

[0101] In the embodiments of the present disclosure, the input word may contain information from at least one modality. "Modality" here refers to different sources or types of data, such as text, images, audio, video, etc. Before the data enters the processing of this method, the raw data from these different modalities will be uniformly converted into standardized numerical vectors through their respective encoders, namely the input word processed by the embodiments of the present disclosure.

[0102] In practical applications, the input data can be of a single modality, such as processing only a piece of text (such as a machine translation task), or processing only one picture (such as an image classification task). More importantly, this method also supports multimodal mixed input, such as processing a video with audio and subtitles. In this case, data from different modalities will be converted into vector words in a unified format by their respective encoders, and then sent to the modality unified Transformer module described in this disclosure for processing. Since the subsequent processing process is independent of the modality type, unified modeling of any modal combination can be achieved.

[0103] In one possible implementation, the process of determining the plurality of first weight parameters by the first gating network is independent of the modal information of the input word.

[0104] In the disclosed embodiment, the first gating network's decision-making process for determining weight parameters is independent of the original modality type of the input word. The gating network's decision is based on the numerical features of the input word vector itself, regardless of whether the vector originally came from text or an image.

[0105] In related technologies, fixed routing expert technology pre-assigns specific experts to different modalities, such as stipulating that text words can only take expert path A, and image words can only take expert path B. This hard-coding method limits the flexibility of the model. For example, if the semantic features of a text word vector representing "car" and an image region vector representing a "car" in an image are close enough in the vector space, the gating network is very likely to assign similar weights to them, thereby routing them to the same or the same group of attention experts for processing. This design truly achieves "equal treatment" of information from different modalities, greatly promotes knowledge sharing and semantic alignment between models in different modalities, and is a key step in realizing a general artificial intelligence model.

[0106] According to the solution of the embodiment of the present disclosure, the gating network does not know whether a word is text or an image. It only analyzes the semantic or structural information contained in this word vector. The benefit of doing so is that the model can discover deep associations between different modal features. For example, an audio word describing "dog barking" and an image word containing "a dog" may have similar vector features at the semantic level, although they come from different modalities. The gating network may assign them to the same expert for processing, and this expert can therefore learn the more abstract concept of "dog" across modalities. This modality-independent, purely feature-based dynamic routing mechanism is the key to the embodiment of the present disclosure being able to effectively promote interaction and fusion between modalities and achieve true modal unification.

[0107] An embodiment of the present disclosure also provides a converter module for modal unification. Figure 2 As shown, the module includes:

[0108] The Mixture of Experts Multi-Head Self-Attention (MoE-MHSA) layer is configured to receive input tokens. The MoE-MHSA layer includes:

[0109] a first gating network, configured to determine routing weights of a plurality of first activatable experts based on vector features of an input word;

[0110] a plurality of first activatable experts, each comprising an independent set of projection parameters for processing an input token according to a routing weight to generate a first set of query, key, and value vectors;

[0111] The first residual connection and layer normalization layer is connected to the MoE-MHSA layer to fuse and normalize the output of the MoE-MHSA layer with the input word to generate an intermediate word;

[0112] A mixture of experts feedforward network (MoE-FFN) layer is configured to receive intermediate word units. The MoE-FFN layer is used to perform nonlinear transformation on the intermediate word units.

[0113] The second residual connection and layer normalization layer is connected to the MoE-FFN layer and is used to fuse and normalize the output of the MoE-FFN layer with the intermediate word to generate the output word.

[0114] In one possible implementation, the MoE-MHSA layer further includes:

[0115] a first shared expert for processing the input token to generate a second set of query, key, and value vectors;

[0116] The MoE-MHSA layer is used to obtain the output of the MoE-FFN layer based on the first set of queries, key and value vectors and the second set of queries, key and value vectors.

[0117] In one possible implementation, the MoE-FFN layer includes:

[0118] a second gating network, configured to determine routing weights of a plurality of second activatable experts based on vector features of the intermediate word units;

[0119] a plurality of second activatable experts for performing nonlinear transformation on the intermediate word units according to the routing weights;

[0120] A second shared expert for performing nonlinear transformation on the intermediate word-units;

[0121] The MoE-FFN layer is used to aggregate the nonlinear transformation results of multiple second activatable experts with the nonlinear transformation results of the second shared expert to obtain the output of the MoE-MHSA layer.

[0122] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0123] Figure 8 Schematic diagram of the structure of the device for data processing with unified modalities provided by the present disclosure. Figure 8 As shown, the apparatus 800 includes:

[0124] The allocation module 801 is used to use the first gating network to determine multiple first weight parameters corresponding to multiple attention experts in the attention layer according to the vector characteristics of the input word.

[0125] The attention module 802 is configured to process the input word-unit based on a plurality of first weight parameters using at least one activated attention expert from a plurality of attention experts to obtain an attention layer output vector.

[0126] The first generation module 803 is used to generate an intermediate word unit based on the attention layer output vector and the input word unit using the first residual connection and the layer normalization layer.

[0127] The second generating module 804 is configured to process the intermediate word-grams using a feed-forward network layer to generate output word-grams.

[0128] In one possible implementation, multiple attention experts each include an independent set of projection parameters for generating query, key, and value vectors to form an independent projection path.

[0129] In one possible implementation, the attention module 802 is configured to:

[0130] Based on a plurality of first weight parameters, at least one activated attention expert among a plurality of attention experts is used to process the input word to obtain a first set of query key-value vectors.

[0131] Based on the first set of query key-value vectors, perform multi-head self-attention calculation to obtain the attention layer output vector.

[0132] In one possible implementation, the attention module 802 is configured to:

[0133] Based on a plurality of first weight parameters, at least one activated attention expert among a plurality of attention experts is used to process the input word to obtain a first set of query key-value vectors.

[0134] The input word is processed using the first shared expert in the attention layer to obtain a second set of query key-value vectors.

[0135] Based on the first set of query key value vectors and the second set of query key value vectors, an attention layer output vector is obtained.

[0136] In one possible implementation, the attention module 802 is configured to:

[0137] Aggregate the first set of query key value vectors and the second set of query key value vectors.

[0138] The aggregation results are subjected to multi-head self-attention calculation to obtain the attention layer output vector.

[0139] In a possible implementation, the second generating module 804 is configured to:

[0140] A second gating network is used to determine a plurality of second weight parameters corresponding to a plurality of feedforward network experts in the feedforward network layer according to the vector features of the intermediate word.

[0141] Based on the multiple second weight parameters, at least one activated feedforward network expert among the multiple feedforward network experts is used to process the intermediate word to obtain a feedforward network expert output vector.

[0142] The output word is generated based on the feedforward network expert output vector and the intermediate word using the second residual connection and layer normalization layer.

[0143] In a possible implementation, the second generating module 804 is configured to determine an activated feedforward network expert from a plurality of feedforward network experts according to a plurality of second weight parameters.

[0144] Intermediate tokens are processed using an activated feedforward network expert.

[0145] The processing results of the activated feedforward network experts are weighted and fused according to a plurality of second weight parameters to obtain a feedforward network expert output vector.

[0146] In a possible implementation, the method further includes:

[0147] The shared expert module is used to process the intermediate word using a second shared expert in the feedforward network layer to obtain a shared expert output vector.

[0148] The second generating module 804 is used for:

[0149] The feedforward network expert output vector and the shared expert output vector are aggregated to obtain a second output vector.

[0150] The second residual connection and the layer normalization layer are used to generate an output word unit according to the second output vector and the intermediate word unit.

[0151] In a possible implementation, the input word contains at least one modality information.

[0152] In one possible implementation, the process of determining the plurality of first weight parameters by the first gating network is independent of the modal information of the input word.

[0153] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0154] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0155] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0156] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0157] like Figure 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0158] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0159] The computing unit 901 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the modality-unified data processing method. For example, in some embodiments, the modality-unified data processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the modality-unified data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the modality-unified data processing method by any other appropriate means (e.g., by means of firmware).

[0160] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0161] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0162] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0164] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0165] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0166] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0167] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A modality-unified data processing method, comprising: Determining, using the first gating network, a plurality of first weight parameters corresponding to a plurality of attention experts in the attention layer according to the vector features of the input word unit; Based on the plurality of first weight parameters, use at least one activated attention expert among the plurality of attention experts to process the input word to obtain an attention layer output vector; Generate an intermediate word based on the attention layer output vector and the input word using a first residual connection and a layer normalization layer; as well as The intermediate word-grams are processed using a feed-forward network layer to generate output word-grams.

2. The method according to claim 1, wherein The multiple attention experts each include an independent set of projection parameters for generating query, key and value vectors to form an independent projection path.

3. The method according to claim 1 or 2, wherein: The step of processing the input word-unit based on the plurality of first weight parameters using at least one activated attention expert among the plurality of attention experts to obtain an attention layer output vector includes: Based on the plurality of first weight parameters, using at least one activated attention expert among the plurality of attention experts to process the input word to obtain a first set of query key-value vectors; Based on the first set of query key-value vectors, perform multi-head self-attention calculation to obtain the attention layer output vector.

4. The method according to claim 1 or 2, wherein: The step of processing the input word-unit based on the plurality of first weight parameters using at least one activated attention expert among the plurality of attention experts to obtain an attention layer output vector includes: Based on the plurality of first weight parameters, using at least one activated attention expert among the plurality of attention experts to process the input word to obtain a first set of query key-value vectors; Processing the input word using a first shared expert in the attention layer to obtain a second set of query key-value vectors; The attention layer output vector is obtained based on the first set of query key value vectors and the second set of query key value vectors.

5. The method according to claim 4, wherein The attention layer output vector is obtained based on the first set of query key value vectors and the second set of query key value vectors, including aggregating the first set of query key value vectors and the second set of query key value vectors; The aggregation result is subjected to multi-head self-attention calculation to obtain the attention layer output vector.

6. The method according to claim 1, wherein The processing of the intermediate word-units by using the feedforward network layer to generate output word-units includes: Determining, using a second gating network, a plurality of second weight parameters corresponding to a plurality of feedforward network experts in the feedforward network layer according to the vector features of the intermediate word; Based on the plurality of second weight parameters, using at least one activated feedforward network expert among the plurality of feedforward network experts to process the intermediate word-gram to obtain a feedforward network expert output vector; An output word-gram is generated based on the feedforward network expert output vector and the intermediate word-gram using a second residual connection and a layer normalization layer.

7. The method according to claim 6, wherein: The step of processing the intermediate word-unit based on the plurality of second weight parameters using at least one activated feedforward network expert among the plurality of feedforward network experts to obtain a feedforward network expert output vector includes: determining an activated feedforward network expert from the plurality of feedforward network experts according to the plurality of second weight parameters; Processing the intermediate word using the activated feedforward network expert; The processing results of the activated feedforward network experts are weighted and fused according to the multiple second weight parameters to obtain a feedforward network expert output vector.

8. The method according to claim 6, further comprising: Processing the intermediate word using a second shared expert in the feedforward network layer to obtain a shared expert output vector; The step of generating an output word-unit based on the feedforward network expert output vector and the intermediate word-unit by using a second residual connection and a layer normalization layer includes: Aggregating the feedforward network expert output vector and the shared expert output vector to obtain a second output vector; An output word-unit is generated according to the second output vector and the intermediate word-unit by utilizing a second residual connection and a layer normalization layer.

9. The method according to any one of claims 1 to 8, wherein: The input word contains at least one modality information.

10. The method according to claim 9, wherein: The process of determining the plurality of first weight parameters by the first gating network is independent of the modality information of the input word.

11. A module for modality-unified data processing, comprising: A mixed-expert multi-head self-attention layer is configured to receive an input word, the mixed-expert multi-head self-attention layer comprising: a first gating network, configured to determine routing weights of a plurality of first activatable experts based on the vector features of the input word; the plurality of first activatable experts each comprising an independent set of projection parameters for processing the input word-grams according to the routing weights to generate a first set of query, key, and value vectors; a first residual connection and layer normalization layer connected to the hybrid expert multi-head self-attention layer, for fusing and normalizing the output of the hybrid expert multi-head self-attention layer with the input word to generate an intermediate word; a hybrid expert feedforward network layer, configured to receive the intermediate word-unit, and the hybrid expert feedforward network layer is used to perform a nonlinear transformation on the intermediate word-unit; The second residual connection and layer normalization layer is connected to the hybrid expert feedforward network layer, and is used to fuse and normalize the output of the hybrid expert feedforward network layer with the intermediate word to generate an output word.

12. The module according to claim 11, wherein The mixed-expert multi-head self-attention layer also includes: a first shared expert for processing the input word to generate a second set of query, key, and value vectors; The hybrid expert multi-head self-attention layer is used to obtain the output of the hybrid expert multi-head self-attention layer based on the first set of queries, key and value vectors and the second set of queries, key and value vectors.

13. The module according to claim 11, wherein The hybrid expert feedforward network layer includes: a second gating network, configured to determine routing weights of a plurality of second activatable experts based on the vector features of the intermediate word; The plurality of second activatable experts are configured to perform nonlinear transformation on the intermediate word-grams according to the routing weights; a second shared expert, configured to perform a nonlinear transformation on the intermediate word; The hybrid expert feedforward network layer is used to aggregate the nonlinear transformation results of the multiple second activatable experts and the nonlinear transformation result of the second shared expert to obtain the output of the hybrid expert feedforward network layer.

14. A device for modality-unified data processing, comprising: an allocation module, configured to determine, using the first gating network, a plurality of first weight parameters corresponding to a plurality of attention experts in the attention layer according to the vector features of the input word unit; an attention module, configured to process the input word-unit using at least one activated attention expert among the plurality of attention experts based on the plurality of first weight parameters to obtain an attention layer output vector; A first generation module is configured to generate an intermediate word according to the attention layer output vector and the input word by using a first residual connection and a layer normalization layer; The second generation module is used to process the intermediate word unit using a feedforward network layer to generate an output word unit.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images