Data processing method, data processing model, and training method therefor

Abstract

To provide a data processing method on natural language processing, deep learning technology, and large-scale language model technology, a data processing model, and a training method therefor.SOLUTION: A data processing method comprises: determining input data containing a plurality of participles; and determining the correlation between each participle of the plurality of participles and each expert network of a plurality of expert networks on the basis of a gating matrix, the plurality of expert networks being used to strengthen the plurality of participles. The data processing method also comprises: distributing the plurality of participles to the plurality of expert networks in a uniform manner on the basis of the correlation and the preset capacity of each expert network; and reinforcing the plurality of participles and determining the data processing results on the basis of the plurality of enhanced participles.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to the field of artificial intelligence, in particular to natural language processing, deep learning technology, large-scale language model technology, in particular to a data processing method, a data processing model and its training method, a data processing device, a training device, an electronic device, a computer-readable storage medium and a computer program product. [Background technology]

[0002] Artificial intelligence is a field that studies how computers can imitate some human thought processes and intelligent behaviors (e.g., learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, artificial intelligence dedicated chips, cloud computing, distributed storage, big data processing, and other technologies, while artificial intelligence software technologies mainly include computer vision technology, voice recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Pre-training language models are a technology that has developed relatively rapidly in the field of natural language processing in recent years. In large-scale language model technology, increasing the size of the pre-training model can generally have a positive effect on the model.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or adopted. Unless otherwise noted, any methods described in this section should not be considered prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise noted, any problem addressed in this section should not be considered an admission of prior art. Summary of the Invention

[0005] The present disclosure provides a data processing method, a data processing model and a training method thereof, a data processing device, a training device, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, a data processing method is provided, the data processing method including: determining input data, the input data including a plurality of participles; determining a correlation between each participle of the plurality of participles and each expert network of a plurality of expert networks based on a gate matrix, where the plurality of expert networks are used to strengthen the plurality of participles; uniformly allocating the plurality of participles to the plurality of expert networks based on the correlation and a preset capacity of each expert network to strengthen the plurality of participles; and determining a data processing result based on the strengthened plurality of participles.

[0007] According to another aspect of the present disclosure, a data processing model is provided, comprising: a routing layer configured to determine a correlation between each participle of a plurality of participles in input data and each expert network of a plurality of expert networks based on a gate matrix, where the correlation is used to uniformly assign the plurality of participles to the plurality of expert networks according to a preset capacity of each expert network; an expert network layer including a plurality of expert networks configured to reinforce the assigned participles; and an output layer configured to determine a data processing result based on the reinforced plurality of participles.

[0008] According to another aspect of the present disclosure, a method for training a data processing model is provided, the method including: determining a sample input, the sample input including a plurality of participles; determining a correlation between each participle of the plurality of participles and each expert network of a plurality of expert networks for enhancing the plurality of participles based on the gate matrix, the correlation being used to uniformly allocate the plurality of participles to the plurality of expert networks according to a preset capacity of each expert network; determining a loss value based on the gate matrix, the closer the gate matrix is ​​to an orthogonal matrix, the smaller the loss value; and adjusting a parameter of the gate matrix based on the loss value to obtain a trained data processing model.

[0009] According to another aspect of the present disclosure, a data processing apparatus is provided, comprising: an input unit configured to determine input data, the input data including a plurality of participles; a correlation determining unit configured to determine a correlation between each participle of the plurality of participles and each expert network of a plurality of expert networks based on a gate matrix, the plurality of expert networks being used to strengthen the plurality of participles; an allocation unit configured to uniformly allocate the plurality of participles to the plurality of expert networks based on the correlations and a preset capacity of each expert network, thereby strengthening the plurality of participles; and an output unit configured to determine a data processing result based on the strengthened plurality of participles.

[0010] According to another aspect of the present disclosure, an apparatus for training a data processing model is provided, the apparatus including: an input unit configured to determine a sample input, the sample input including a plurality of participles; a correlation determining unit configured to determine a correlation between each participle of the plurality of participles and each expert network of a plurality of expert networks for enhancing the plurality of participles based on the gating matrix, the correlation being used to uniformly allocate the plurality of participles to the plurality of expert networks according to a preset capacity of each expert network; a loss determining unit configured to determine a loss value based on the gating matrix, the closer the gating matrix is ​​to an orthogonal matrix, the smaller the loss value; and a parameter adjusting unit configured to adjust a parameter of the gating matrix based on the loss value to obtain a trained data processing model.

[0011] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided in the present disclosure.

[0012] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to cause the computer to perform a method provided by the present disclosure.

[0013] According to another aspect of the present disclosure there is provided a computer program product including a computer program, which when executed by a processor implements a method provided by the present disclosure.

[0014] According to one or more embodiments of the present disclosure, when allocating data processing tasks to multiple expert networks, the uniform allocation constraint can be satisfied while considering the optimal allocation, thereby allowing each expert network to function in the data processing process and improving data processing efficiency.

[0015] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, and are not intended to limit the scope of protection of the present disclosure. Other features of the present disclosure will be easily understood from the following description. [Brief description of the drawings]

[0016] The drawings illustratively illustrate the examples, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the examples. The illustrated examples are for illustrative purposes only and are not intended to limit the scope of the claims. In all the drawings, the same reference numerals refer to similar, but not necessarily identical, elements.

[0017] [Figure 1] 1 shows a schematic diagram of an example system capable of implementing various methods described herein, according to embodiments of the present disclosure. [Diagram 2] 1 illustrates an exemplary process of a data processing method according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 illustrates an exemplary block diagram of a data processing model according to an embodiment of the present disclosure. [Figure 4] 1 illustrates an example flow chart of a method for training a data processing model according to an embodiment of the present disclosure. [Figure 5A] 1 shows an example of a data processing process according to the related art. [Figure 5B] 1 illustrates an example of a data processing process according to an embodiment of the present disclosure. [Figure 6] 1 illustrates an exemplary block diagram of a data processing apparatus according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates an exemplary block diagram of an apparatus for training a data processing model according to an embodiment of the present disclosure. [Figure 8] FIG. 1 illustrates a block diagram of an exemplary electronic device that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings, and various details in the embodiments of the present disclosure contained therein are to be considered as merely illustrative, in order to facilitate understanding. Therefore, it should be recognized that those skilled in the art can make various changes and modifications to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the following description omits descriptions of known functions and structures.

[0019] In this disclosure, unless otherwise specified, the terms "first," "second," and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of an element, or in some cases, may refer to different instances based on the contextual description.

[0020] The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, an element may be one or more, unless specifically limited in number. It should be noted that the term "and / or" as used in the present disclosure covers any and all possible combinations of the listed items.

[0021] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. 1 illustrates a schematic diagram of an example system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. With reference to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0022] In an embodiment of the present disclosure, the server 120 operates to perform one or more services or software applications of methods according to an embodiment of the present disclosure.

[0023] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software-as-a-Service (SaaS) model.

[0024] In the configuration shown in FIG. 1, server 120 may include one or more assemblies that implement the functions performed by server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof that may be executed on one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn use one or more client applications to interact with server 120 to utilize the services provided by these assemblies. It should be understood that a variety of different system configurations are possible and may differ from system 100. Thus, FIG. 1 is an example of a system for implementing various methods described herein and is not intended to be limiting.

[0025] A user may use client devices 101, 102, 103, 104, 105, and / or 106 to obtain data to be processed by a method according to an embodiment of the present disclosure. The client devices may provide an interface through which a user of the client device interacts with the client device. The client devices may also output information to the user through the interface. Although only six client devices are shown in FIG. 1, one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0026] The client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (e.g., GOOGLE Chrome OS), and may include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android, and the like. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), and the like. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), Short Message Service (SMS) applications, and may use a variety of communication protocols.

[0027] Network 110 may be any type of network known to those skilled in the art that may use any one of a number of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0028] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain the server's virtual storage devices). In various embodiments, server 120 may execute one or more services or software applications that provide the functionality described below.

[0029] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercially available server operating system. Server 120 may also run any one of a variety of additional server applications and / or mid-tier applications, such as an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0030] In some embodiments, server 120 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0031] In some embodiments, the server 120 may be a server of a distributed system or a server incorporating blockchain. The server 120 may be a cloud server, or an intelligent cloud computing server or an intelligent cloud host equipped with artificial intelligence technology. The cloud server is a host product in a cloud computing service system, and solves the defects of the traditional physical host and virtual private server (VPS) services, such as high management difficulty and low business scalability.

[0032] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files or video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of a variety of types. In some embodiments, a database used by server 120 may be a relational database. One or more of these databases may store, update, and retrieve data from the database and from the database in response to instructions.

[0033] In some embodiments, one or more of databases 130 may be used by an application to store data for the application. The databases used by the application may be various types of databases, such as a key-value repository, an object repository, a general-purpose repository supported by a file system, etc.

[0034] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the application of the various methods and apparatus described in accordance with this disclosure. FIG. 2 illustrates an exemplary process of a data processing method according to an embodiment of the present disclosure.

[0035] As shown in FIG. 2, in step S202, input data can be determined, where the input data includes multiple participles. In step S204, a correlation between each participle of the plurality of participles and each expert network of the plurality of expert networks can be determined based on the gate matrix, where the plurality of expert networks are used to enhance the plurality of participles.

[0036] In step S206, the participles can be strengthened by uniformly allocating the participles to the expert networks based on the correlation and the preset capacity of each expert network.

[0037] In step S208, the data processing result can be determined based on the enriched plural participles. According to the embodiments of the present disclosure, when allocating data processing tasks to multiple expert networks, the uniform allocation constraint can be satisfied while considering the optimal allocation, thereby allowing each expert network to play a role in the data processing process and improving the data processing efficiency.

[0038] The principles of the present disclosure will now be described in detail. The Transformer model is a neural network model based on attention mechanism, which is mainly used for natural language processing (NLP) tasks such as language translation, text classification, etc. The Transformer architecture is widely used in the field of natural language processing. The Transformer model mainly consists of two parts: an encoder and a decoder. The encoder converts the input sequence into a fixed-length vector representation, and the decoder converts the vector into an output sequence. Between the encoder and the decoder, there are one or more attention mechanism layers that capture the correlation between the input sequence and the output sequence. Specifically, the Transformer model is mainly composed of multiple TransformerLayers, and each TransformerLayer contains two main parts: a self-attention layer and a feed-forward network (FFN). The self-attention mechanism is a method for calculating and integrating the correlation between each word and other words. This method captures the relationship between each element in the input sequence, regardless of the distance in the sequence. In the Transformer model, the self-attention mechanism is used to calculate the influence of each element on other elements in the input sequence or output sequence. The feed-forward neural network is a neural network based on a fully connected layer, which is used for nonlinear transformation of the input sequence. In the Transformer model, the FFN can consist of two fully connected layers.

[0039] In the current Transformer network structure, a large number of matrix multiplications are generally performed, and the size of the network parameters is directly proportional to the number of matrix operations, which leads to a series of problems in practical operation, such as:

[0040] 1. High computational complexity. Transformer-based models have complex structures and involve a large amount of matrix multiplication and attention mechanism calculations, resulting in long computation and training times. This is a significant bottleneck for large-scale datasets and real-time applications.

[0041] 2. High resource consumption. Transformer-based models have high computational complexity, so their training and inference processes require a large amount of computing resources, including GPUs, TPUs, etc. This is a challenge for resource-limited environments.

[0042] 3. Limited processing ability for long sequences. Transformer-based models are good at processing short sequences of data, but when processing long sequences of data, performance degradation may become an issue. This is mainly because the self-attention mechanism of the model needs to calculate a large number of attention points when processing long sequences, which reduces computational efficiency.

[0043] Therefore, a Transformer network structure requires a large number of parameters to achieve optimal results: doubling the amount of model parameters means doubling the computational resources in traditional structures.

[0044] The data processing model in this disclosure relates to a Mix of Experts (MoE) network in a large-scale pre-training model (e.g., large-scale language model, big language model). In simple terms, the MoE model increases the number of Feed-Forward Network (FFN) modules in the Transformer architecture, and each Feed-Forward Network is an expert. In the data processing process, different participles (also called tokens or symbols, tokens) input into the model are selectively assigned to different experts for calculation to realize the enhancement of participles (e.g., feature vectors that characterize participles). In this way, the amount of parameters of the model can be amplified (e.g., when Z experts are set, the parameters of the FFN module can be amplified by Z times), but the amount of calculation is the same as before.

[0045] The advantage of the mixed expert network is that it allows the decoupling of computation and storage, which means that computation resources and storage capacity can be scaled and optimized independently. With this characteristic, the mixed expert network provided by the embodiment of the present disclosure can accommodate more model parameters within limited computation resources, thereby improving the expressiveness and performance of the model. In addition, this decoupling design can flexibly adjust the allocation of computation resources and storage resources according to actual needs to achieve optimal performance and efficiency. Therefore, the mixed expert network provided by the embodiment of the present disclosure provides an effective solution for efficient utilization of computation resources, and can exert greater advantages, especially in resource-limited environments.

[0046] In step S202, input data may be determined, the input data including multiple participles. In a text processing example, a text participle algorithm can be used to process a natural language input to obtain corresponding participles. The participles of the input data may include tokens that correspond to single characters or words of the input text, or may include tokens that correspond to special symbols. Special symbols can include, for example, a sentence-initial symbol [CLS] and a symbol [SEP] that is used to split different sentences or is located at the end of a sentence.

[0047] When using Transformer-based network structures in other data processing fields, participles can correspondingly represent divided data segments in other data (e.g., images, etc.).

[0048] In step S204, a correlation between each participle of the plurality of participles and each expert network of the plurality of expert networks can be determined based on the gate matrix, where the plurality of expert networks are used to enhance the plurality of participles.

[0049] In the MoE model, a routing module is used to assign the input participle to a corresponding expert network. Here, the routing module corresponds to a first-order projection, and projects the vector representation of the input participle to each expert network. The routing module can be expressed as Equation (1).

[0050]

number

[0051] Here, X represents the input data, Linear represents the linear mapping function, and softmax represents normalizing the result of the linear mapping. The results output by the routing module can be used to represent the correlation between each participle and each of the multiple expert networks.

[0052] TIFF2024170549000003.tif87170

[0053]

number

[0054] Here, X represents the input data and W represents the gate matrix. In step S206, the participles can be strengthened by uniformly allocating the participles to the expert networks based on the correlation and the preset capacity of each expert network.

[0055] In the prior art, in the forward process of each network, the participle is processed by the expert network that has the highest correlation with the participle, which is determined in the routing module. However, such an allocation method causes a phenomenon called "model collapse", and in the case of multiple newly introduced experts, the model tends to use only one or two of them, and the remaining "expert" networks cannot be effectively trained and used, so that the prediction effect of the network cannot be effectively improved. In the related art, induction using an auxiliary loss function makes it possible to train each "expert" network at the same time. The auxiliary loss function generally has the form of the following formula (3):

[0056]

number

[0057] By increasing the entropy of the gate for prediction, the prediction results by the routing module are induced to move toward the average. However, even when induced using an auxiliary loss function, in actual network operation, multiple tokens need to perform scoring predictions in parallel to ensure high computational performance. On the other hand, when multiple tokens perform "routing" in parallel, "gathering in one place" occurs, and the number of tokens obtained by each "expert" does not match. In modern computing hardware, this mismatch leads to reduced computational efficiency.

[0058] To solve the above problems, the present disclosure assigns multiple participles to different expert networks uniformly, so that all expert networks use computing resources in a uniform manner, improving computational efficiency.

[0059] Based on the number of participles in the input data and the number of expert networks, a predetermined capacity of each expert network, i.e., the number of participles each expert network can process, can be determined. For example, the predetermined capacity of each expert network may be determined as M / e, where M is the number of participles in the input data and e is the number of expert networks. In some examples, the result of M / e may be rounded up or down to determine the predetermined capacity of each expert network. Each time a participle is assigned to an expert network, it occupies a portion of the capacity of that expert network.

[0060] In some embodiments, for each participle, the expert network with the highest correlation with the participle can be determined using the correlation determined based on the gate matrix. If the remaining capacity of the expert network with the highest correlation is not zero, the participle can be assigned to an expert network that still has remaining capacity. Correlations can be calculated for multiple participles in turn and assigned to expert networks.

[0061] In another embodiment, multiple allocations can be made for multiple participles. Each allocation makes the expert network strengthen multiple input participles at once. The number of allocations can be set to adjust the computation amount of the network for the input participles. Obviously, the computation resources that need to be occupied by the data processing model according to the embodiment of the present disclosure are determined based on the number of input participles and the number of times the expert network processes the participles, while the amount of parameters that the data processing model stores is determined based on the number of expert networks. There is no correlation between the two. Therefore, the decoupling of computation amount and memory amount can be realized.

[0062] In the case of multiple allocation, some of the allocation plans can assign each participle to the expert network with which it has the highest correlation, while other allocation plans can be constrained to ensure that the final distribution is a uniform allocation.

[0063] For example, multiple participles can be allocated twice, where in the first allocation, each participle can be assigned to whichever expert network is most highly correlated with the participle. In the second allocation, each participle is assigned to the most relevant expert network possible, subject to the uniform allocation constraint. The result of the two allocations is that each expert network processes roughly the same number of participles, allowing for uniform allocation of computational resources. In some examples, the uniform allocation can be achieved based on an optimal transportation algorithm.

[0064] An optimal transportation algorithm is an algorithm used to solve transportation problems in the fields of mathematics and optimization. In its most basic form, it concerns how to move goods from one set of locations to another set of locations at the least cost. There are many variations of this problem, including in fields such as logistics, image processing, and machine learning. In an optimal transportation algorithm, there is typically a "cost matrix" whose elements represent the cost of moving from one location to another. The goal of the algorithm is to find a transportation solution that minimizes the total transportation cost while satisfying all demand and supply constraints. To solve this problem, optimal transportation algorithms typically combine mathematical tools such as linear programming and graph theory. Large-scale problems also require more efficient computational skills and optimization techniques. In the context of machine learning and deep learning, optimal transportation algorithms are used as a tool to improve the performance of models, especially when it comes to alignment, mapping, or transfer learning between data distributions. This helps to establish better correspondence between different data sets, which can improve the generalization ability of the model. Optimal transportation algorithms can be used to achieve highly efficient, low-cost resource allocation and transmission solutions in various application scenarios.

[0065] The assignment problem for participles in the embodiments of the present disclosure can be transformed into a transportation problem, whereby an optimal transportation algorithm can be utilized to achieve a uniform assignment of participles to each expert network.

[0066] In some examples, step S206 may include: determining a first allocation plan based on the correlation, in which each participle is assigned to an expert network with the highest correlation with the participle; determining a remaining capacity of each expert network based on the first allocation plan and a preset capacity of each expert network; and determining a second allocation plan based on the correlation and the remaining capacity of each expert network, in which the second allocation plan combines the first allocation plan such that the participles are uniformly assigned to the expert networks. For example, the second allocation plan may be determined based on the correlation and the remaining capacity of each expert network based on an optimal transportation algorithm. In an example, the optimal transportation algorithm may be a sinkhorn algorithm or any other suitable optimal transportation algorithm. The sinkhorn algorithm may take into account the cost (correlation between the participle and the expert network) to satisfy the uniform allocation constraint. The sinkhorn algorithm may be used to efficiently implement acceleration hardware such as a GPU. By transforming the process of allocating participles to a network of experts into an optimal transportation problem, an optimal transportation algorithm is used to have each participle select an expert that is most highly correlated with itself as possible, while ensuring uniformity of the transportation results, so that each expert processes the same (or approximately the same) number of participles, achieving optimal computational efficiency.

[0067] In some examples, the preset capacity of each expert network may be the same. When multiple allocations are performed, the preset capacity of each expert network can be determined based on (N×M) / e, where N is the number of allocations, M is the number of participles in the input data, and e is the number of expert networks.

[0068] In step S208, a data processing result can be determined based on the reinforced participles. For example, the expert network realizes the function of FFN in the Transformer network architecture. After the assigned participles are reinforced through the expert network, the reinforced participles can be input to a next layer in the network, such as a self-attention layer. In some examples, step S208 can include processing the reinforced participles based on a self-attention mechanism to obtain a data processing result.

[0069] FIG. 3 illustrates an exemplary block diagram of a data processing model according to an embodiment of the present disclosure. The data processing method illustrated in FIG. 2 can be implemented using the data processing model illustrated in FIG. 3. The data processing model described in connection with FIG. 3 may be a deep neural network or any other mathematical model consistent with the principles of the present disclosure. The structure of the data processing model illustrated in FIG. 3 can be used for large-scale language models such as big language models.

[0070] As shown in FIG. 3, the data processing model can include a routing layer 310 , an expert network layer 320 that includes multiple expert networks, and an output layer 330 .

[0071] The routing layer 310 is configured to determine a correlation between each participle of the plurality of participles in the input data and each expert network of the plurality of expert networks based on a gate matrix, where the correlation is used to uniformly allocate the plurality of participles to the plurality of expert networks according to a preset capacity of each expert network.

[0072] An expert network in the expert network layer 320 can be configured to enforce the assigned participles. The output layer 330 can be configured to determine a data processing result based on the enriched plurality of participles.

[0073] In some embodiments, determining a correlation between each participle of the plurality of participles and each expert network of the plurality of expert networks based on a gate matrix can include determining a correlation between each expert network of the plurality of expert networks and each participle of the plurality of participles based on a product of the input data and the gate matrix.

[0074] In some embodiments, evenly allocating the plurality of participles to the plurality of expert networks may include determining a first allocation scheme based on the correlations, where in the first allocation scheme, each participle is assigned to an expert network with which it has a highest correlation; determining a remaining capacity of each expert network based on the first allocation scheme and a preset capacity of each expert network; and determining a second allocation scheme based on the correlations and the remaining capacity of each expert network, where the second allocation scheme is combined with the first allocation scheme such that the plurality of participles are evenly allocated to the plurality of expert networks.

[0075] In some embodiments, the preset capacity of each expert network may be the same. In some embodiments, determining the second allocation plan based on the correlation and the remaining capacity of each expert network can include determining the second allocation plan based on the correlation and the remaining capacity of each expert network based on an optimal input algorithm.

[0076] In some embodiments, the optimal transportation algorithm may be a sinkhorn algorithm. In some embodiments, the output layer may be configured to process the multiple participles enhanced based on the self-attention mechanism to obtain a data processing result.

[0077] 4 shows an exemplary flowchart of a method for training a data processing model according to an embodiment of the present disclosure. The method shown in FIG. 4 can be used to train the data processing model described in relation to FIG.

[0078] In step S402, a sample input may be determined, where the sample input includes multiple participles. Content of the sample input may be selected from a pre-determined training data set.

[0079] In step S404, a correlation between each participle of the input participles and each expert network of the multiple expert networks can be determined based on the gate matrix, where the multiple expert networks are used to strengthen the multiple participles. The correlation determined in step S204 can be used to uniformly allocate the multiple participles to the multiple expert networks according to the preset capacity of each expert network, where the preset capacity of each expert network is the same, and the method described in connection with FIG. 2 in this disclosure can be used to determine the preset capacity of each expert network.

[0080] In step S406, a loss value can be determined based on a gate matrix. Here, the closer the gate matrix is ​​to an orthogonal matrix, the smaller the loss value is. In some embodiments, the loss value can be designed based on the characteristics of an orthogonal matrix. One of the characteristics of an orthogonal matrix is ​​that the product of an orthogonal matrix and its transpose matrix is ​​an identity matrix. Therefore, the product of a gate matrix and its transpose matrix can be determined, and the loss value can be determined based on the error with the identity matrix. The loss value can be determined, for example, based on the following formula (4).

[0081]

number

[0082] TIFF2024170549000007.tif36170

[0083] In step S408, the parameters of the gating matrix can be adjusted based on the loss value determined in step S406 to obtain a trained data processing model.

[0084] According to the model training method provided by the embodiment of the present disclosure, by setting the loss value so that the vectors corresponding to each expert network in the gate matrix representing the trainable parameters of each expert network in the trained data processing model are (almost) orthogonal to each other, each expert network can enhance the participles in a different way. When the gate matrix is ​​close to an orthogonal matrix, each expert network correlates with each participle in a different way, so that each expert network tends to select different participles, and when assigning an expert network to each participle, the expert networks tend to select the same participle, thereby avoiding the situation where partial participles are missing.

[0085] Fig. 5A shows an example of a data processing process according to the related art. In the example shown in Fig. 5A, there exists a phenomenon that the parameters of the routing module are homogenized, specifically, the vector square angles for each expert network in the gate matrix are close, as shown in formula (5).

[0086]

number

[0087] TIFF2024170549000009.tif28170

[0088] In the example shown in FIG. 5A, due to the presence of homogenization phenomenon, different expert networks 510 and 520 tend to select the same participles (e.g., C, D, E, F, G) for processing, whereby partial participles (A, B) are overlooked and cannot be processed.

[0089] 5B shows an example of a data processing process according to an embodiment of the present disclosure. Since the vectors for each expert network in the gate matrix in the data processing model trained by the training method described with reference to FIG. 4 tend to be orthogonal, different expert networks can select different participles to process, avoiding the homogenization problem seen in FIG. 5A.

[0090] 6 shows an exemplary block diagram of a data processing apparatus according to an embodiment of the present disclosure. As shown in FIG. 6, the data processing apparatus 600 may include an input unit 610, a correlation determining unit 620, an allocation unit 630, and an output unit 640. The data processing apparatus illustrated in FIG. 6 may be used to execute the data processing method described in relation to FIG. 2.

[0091] The input unit 610 is configured to determine input data, the input data including a plurality of participles. The correlation determining unit 620 is configured to determine a correlation between each participle of the plurality of participles and each expert network of the plurality of expert networks based on a gate matrix, where the plurality of expert networks are used to enhance the plurality of participles.

[0092] The allocation unit 630 is configured to uniformly allocate the plurality of participles to the plurality of expert networks based on correlations and preset capacities of each expert network, and reinforce the plurality of participles.

[0093] The output unit 640 is configured to determine a data processing result based on the plurality of emphasized participles. In some embodiments, the correlation determining unit is configured to determine a correlation between each expert network of the plurality of expert networks and each participle of the plurality of participles based on a product of the input data and a gate matrix.

[0094] In some embodiments, evenly allocating the plurality of participles to the plurality of expert networks based on the correlations and the preset capacity of each expert network can include determining a first allocation plan based on the correlations, where in the first allocation plan, each participle is assigned to an expert network with which it has a highest correlation; determining a remaining capacity of each expert network based on the first allocation plan and the preset capacity of each expert network; and determining a second allocation plan based on the correlations and the remaining capacity of each expert network, where the second allocation plan is combined with the first allocation plan such that the plurality of participles are evenly allocated to the plurality of expert networks.

[0095] In some embodiments, the preset capacity of each expert network may be the same. In some embodiments, determining the second allocation plan based on the correlation and the remaining capacity of each expert network can include determining the second allocation plan based on the correlation and the remaining capacity of each expert network based on an optimal input algorithm.

[0096] In some embodiments, the optimal transportation algorithm may be a sinkhorn algorithm. In some examples, determining the data processing result based on the enhanced plurality of participles may include processing the enhanced plurality of participles based on a self-attention mechanism to obtain the data processing result.

[0097] It should be understood that each module or unit of the apparatus 600 shown in Fig. 6 may correspond to each step of the method 200 described with reference to Fig. 2. Thus, the operations, features, and advantages described with respect to the method 200 described above are equally applicable to the apparatus 600 and the modules and units included in the apparatus 600. For the sake of brevity, some operations, features, and advantages will not be described here.

[0098] Fig. 7 illustrates an exemplary block diagram of an apparatus for training a data processing model according to an embodiment of the present disclosure. As shown in Fig. 7, the apparatus 700 may include an input unit 710, a correlation determination unit 720, a loss determination unit 730, and a parameter adjustment unit 740. The data processing apparatus illustrated in Fig. 7 may be used to execute the method for training a data processing model described in relation to Fig. 4.

[0099] The input unit 710 may be configured to determine a sample input, the sample input including a plurality of participles. The correlation determination unit 720 can be configured to determine a correlation between each participle of the plurality of participles and each expert network of the plurality of expert networks based on a gate matrix, where the plurality of expert networks are used to strengthen the plurality of participles, and the correlation is used to uniformly allocate the plurality of participles to the plurality of expert networks according to a preset capacity of each expert network.

[0100] The loss determination unit 730 may be configured to determine the loss value based on a gating matrix, where the closer the gating matrix is ​​to an orthogonal matrix, the smaller the loss value.

[0101] The parameter adjustment unit 740 may be configured to adjust parameters of the gating matrix based on the loss value to obtain a trained data processing model.

[0102] In some embodiments, the loss determination unit may be configured to determine a product of a gate matrix and a transpose of the gate matrix, and determine a loss value based on an error between the product and an identity matrix.

[0103] In some embodiments, the error is the minimum mean square error. It should be understood that each module or unit of the apparatus 700 shown in Figure 7 may correspond to each step of the method 400 described with reference to Figure 4. Thus, the operations, features, and advantages described with respect to the method 400 described above are equally applicable to the apparatus 700 and the modules and units included in the apparatus 700. For the sake of brevity, some operations, features, and advantages will not be described here.

[0104] While particular functions have been discussed above with reference to particular modules, it should be noted that the functionality of each unit described herein may be split into multiple units and / or at least some of the functionality of multiple units may be combined into a single unit.

[0105] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Each unit described with respect to FIG. 6, FIG. 7 may be realized in hardware or in hardware in combination with software and / or firmware. For example, these units may be implemented as computer program code / instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units may be implemented as hardware logic / circuitry. For example, in some embodiments, one or more of units 610-640, units 710-740 may be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (e.g., a processor (e.g., including a Central Processing Unit (CPU), microcontroller, microprocessor, Digital Signal Processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components in other circuits) that may optionally execute received program code and / or include embedded firmware to perform functions.

[0106] According to another aspect of the present disclosure, there is further provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions, when executed by the at least one processor, enabling the at least one processor to perform a method according to an embodiment of the present disclosure.

[0107] According to another aspect of the present disclosure, there is also provided a non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions being used to cause said computer to perform a method according to an embodiment of the present disclosure.

[0108] According to another aspect of the present disclosure, there is also provided a computer program product including a computer program, which when executed by a processor, implements a method according to an embodiment of the present disclosure.

[0109] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and morals.

[0110] According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are further provided. Next, referring to FIG. 8, a configuration block diagram of an electronic device 800 that can function as a server or client of the present disclosure will be described, which may be an example of a hardware device applicable to each aspect of the present disclosure. The electronic device may represent various forms of digital electronic computers, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may further represent various forms of mobile devices, such as personal digital processing, mobile phones, intelligent phones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure described and / or claimed herein.

[0111] 8, the electronic device 800 includes a computing unit 801 that can perform various suitable operations and processes based on a program stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 further stores various programs and data required for operating the electronic device 800. The computing unit 801, the ROM 802 and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0112] A number of components in the electronic device 800 are connected to the I / O interface 805, and the number of components include an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 may be any type of device capable of inputting information to the electronic device 800, and the input unit 806 may receive input numeric or character information and generate key signal inputs for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote control. The output unit 807 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, or a printer. The storage unit 808 may include, but is not limited to, a magnetic disk, an optical disk. The communication unit 809 enables the electronic device 800 to exchange information / data with other devices via computer networks and / or various telecommunication networks, such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as, for example, Bluetooth. TM devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0113] The computing unit 801 may be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 executes various methods and processes, such as the methods 200, 400 described above. For example, in some embodiments, the methods 200, 400 can be realized as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, some or all of the computer program can be loaded and / or installed in the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the methods 200, 400 described above can be executed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the methods 200, 400 in any other suitable manner (eg, using firmware).

[0114] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may receive data and instructions from a storage system, at least one input device, and at least one output device, and may transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] The program code implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, so that when executed by the processor or controller, the program code performs the functions / operations specified in the flow charts and / or block diagrams. The program code may be entirely executed on a machine, partly executed on a machine, partly executed on a machine as a separate software package and partly executed on a remote machine, or entirely executed on a remote machine or server.

[0116] In the context of the present disclosure, a machine-readable medium may be a tangible medium, which may include or store a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection with one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0117] To provide for interaction with a user, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user may provide input to the computer. Other types of devices may also be provided for providing interaction with a user, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback) and may receive input from the user in any form (including sound input, speech input, or tactile input).

[0118] The systems and techniques described herein may be implemented in a computing system including a backstage component (e.g., as a data server), a middleware component (e.g., an application server), a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique embodiment), or any combination of the backstage component, middleware component, or front-end component. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0119] The computer system may include a client side and a server. The client side and the server are generally remote from each other and usually interact with each other via a communication network. The relationship between the client side and the server is generated by running computer programs having a client side-server relationship with each other on corresponding computers. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0120] It should be understood that the steps may be reordered, added, or removed using the various forms of flow described above. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, and the present disclosure is not limited thereto as long as the technical solutions disclosed in this disclosure can achieve the desired results.

[0121] Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but only by the scope of the appended claims and their equivalents. Various elements of the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in a different order from the order described in this disclosure. Furthermore, various elements of the embodiments or examples may be combined in various ways. It is important to note that with the evolution of technology, many elements described herein may be replaced by equivalent elements appearing later in this disclosure.

Claims

1. 1. A data processing method comprising the steps of: determining input data, the input data including a plurality of participles; determining a correlation between each participle of the plurality of participles and each expert network of a plurality of expert networks based on a gate matrix, and the plurality of expert networks are used to enhance the plurality of participles; uniformly allocating the plurality of participles to the plurality of expert networks based on the correlations and a preset capacity of each expert network to consolidate the plurality of participles; determining a data processing result based on the enriched plurality of participles.

2. determining a correlation between each participle of the plurality of participles and each expert network of the plurality of expert networks based on a gate matrix, 2. The data processing method of claim 1, further comprising determining a correlation between each of said plurality of expert networks and each participle of said plurality of participles based on a product of said input data and said gate matrix.

3. uniformly allocating the plurality of participles to the plurality of expert networks based on the correlations and a preset capacity of each expert network, determining a first assignment scheme based on the correlations, wherein each participle is assigned to the expert network with which it has a highest correlation; determining a remaining capacity of each expert network based on the first allocation plan and a preset capacity of each expert network; determining a second allocation plan based on the correlation and the remaining capacity of each of the expert networks, and combining the second allocation plan with the first allocation plan such that the participles are evenly allocated to the expert networks.

4. 4. The data processing method of claim 3, wherein the preset capacity of each of said expert networks is the same.

5. determining a second allocation plan based on the correlation and the remaining capacity of each of the expert networks; 4. The method of claim 3, further comprising: determining a second allocation plan based on said correlations and the remaining capacity of each of said expert networks based on an optimal transportation algorithm.

6. 6. The data processing method of claim 5, wherein said optimal transportation algorithm is the Sinkhorn algorithm.

7. determining a data processing result based on the enriched plurality of participles, The data processing method according to any one of claims 1 to 6, comprising processing the enhanced plurality of participles based on a self-attention mechanism to obtain the data processing result.

8. A data processing model, a routing layer configured to determine a correlation between each participle of the plurality of participles in the input data and each expert network of the plurality of expert networks based on a gate matrix, the correlation being used to uniformly allocate the plurality of participles to the plurality of expert networks according to a preset capacity of each expert network; an expert network layer including a plurality of expert networks configured to reinforce the assigned participles; and an output layer configured to determine a data processing result based on the enriched plurality of participles.

9. determining a correlation between each participle of the plurality of participles and each expert network of the plurality of expert networks based on a gate matrix, 9. The data processing model of claim 8, further comprising determining a correlation between each of said plurality of expert networks and each participle of said plurality of participles based on a product of said input data and said gate matrix.

10. uniformly allocating the plurality of participles to the plurality of expert networks determining a first assignment scheme based on the correlations, wherein each participle is assigned to the expert network with which it has a highest correlation; determining a remaining capacity of each expert network based on the first allocation plan and a preset capacity of each expert network; determining a second allocation plan based on the correlation and the remaining capacity of each of the expert networks, and combining the second allocation plan with the first allocation plan such that the participles are evenly allocated to the expert networks.

11. 11. The data processing model of claim 10, wherein the preset capacity of each of the expert networks is the same.

12. determining a second allocation plan based on the correlation and the remaining capacity of each of the expert networks; 11. The data processing model of claim 10, further comprising determining a second allocation plan based on said correlation and the remaining capacity of each of said expert networks based on an optimal transportation algorithm.

13. 13. The data processing model of claim 12, wherein the optimal transportation algorithm is the Sinkhorn algorithm.

14. The output layer is The data processing model according to any one of claims 8 to 13, configured to process the enriched plurality of participles based on a self-attention mechanism to obtain the data processing result.

15. A method for training a data processing model according to any one of claims 1 to 14, comprising: determining a sample input, the sample input including a plurality of participles; determining a correlation between each participle of the plurality of participles and each expert network of a plurality of expert networks for enhancing the plurality of participles according to the gate matrix, and the correlation is used to uniformly allocate the plurality of participles to the plurality of expert networks according to a preset capacity of each expert network; determining a loss value based on the gate matrix, the closer the gate matrix is ​​to an orthogonal matrix, the smaller the loss value; and adjusting parameters of the gating matrix based on the loss value to obtain a trained data processing model.

16. Determining a loss value based on the gate matrix includes: determining a product of the gate matrix and a transpose of the gate matrix; and determining the loss value based on an error between the product and an identity matrix.

17. The method of claim 16, wherein the error is a minimum mean square error.

18. 1. A data processing device, comprising: an input unit configured to determine input data, the input data including a plurality of participles; a correlation determination unit configured to determine a correlation between each participle of the plurality of participles and each expert network of a plurality of expert networks based on a gate matrix, the plurality of expert networks being used to enhance the plurality of participles; an allocation unit configured to uniformly allocate the plurality of participles to the plurality of expert networks based on the correlations and a preset capacity of each expert network to reinforce the plurality of participles; and an output unit configured to determine a data processing result based on the enriched plurality of participles.

19. The correlation determination unit includes:

20. The data processing apparatus of claim 18, further configured to determine a correlation between each of said plurality of expert networks and each participle of said plurality of participles based on a product of said input data and said gate matrix.

20. uniformly allocating the plurality of participles to the plurality of expert networks based on the correlations and a preset capacity of each expert network, determining a first assignment scheme based on the correlations, wherein each participle is assigned to the expert network with which it has a highest correlation; determining a remaining capacity of each expert network based on the first allocation plan and a preset capacity of each expert network; determining a second allocation plan based on the correlation and the remaining capacity of each of the expert networks, and combining the second allocation plan with the first allocation plan such that the participles are evenly allocated to the expert networks.

21. 21. A data processing apparatus as claimed in claim 20, wherein the preset capacity of each of said expert networks is the same.

22. determining a second allocation plan based on the correlation and the remaining capacity of each of the expert networks; 21. The data processing apparatus of claim 20, further comprising determining a second allocation plan based on said correlations and the remaining capacity of each of said expert networks based on an optimal transportation algorithm.

23. 23. A data processing apparatus as claimed in claim 22, wherein said optimal transport algorithm is the Sinkhorn algorithm.

24. determining a data processing result based on the enriched plurality of participles, A data processing apparatus according to any one of claims 18 to 23, configured to process the enhanced plurality of participles based on a self-attention mechanism to obtain the data processing result.

25. An apparatus for training a data processing model according to any one of claims 1 to 14, comprising: a sample input configured to determine a sample input, the sample input including an input unit including a plurality of participles; a correlation determining unit, configured to determine a correlation between each participle of the plurality of participles and each expert network of a plurality of expert networks for enhancing the plurality of participles according to the gate matrix, the correlation being used to uniformly allocate the plurality of participles to the plurality of expert networks according to a preset capacity of each expert network; a loss determination unit configured to determine a loss value based on the gate matrix, the closer the gate matrix is ​​to an orthogonal matrix, the smaller the loss value; and a parameter adjusting unit configured to adjust parameters of the gating matrix based on the loss value to obtain a trained data processing model.

26. The loss determination unit includes: determining a product of the gate matrix and a transpose of the gate matrix; The apparatus of claim 25 , configured to determine the loss value based on an error between the product and an identity matrix.

27. 27. The apparatus of claim 26, wherein the error is a minimum mean square error.

28. An electronic device, At least one processor; a memory communicatively coupled to the at least one processor; An electronic device, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method according to any one of claims 1 to 7 and 15 to 17.

29. A non-transitory computer readable storage medium having stored thereon computer instructions, the computer instructions being used to cause the computer to carry out the method of any one of claims 1 to 7, 15 to 17.

30. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7, 15 to 17.

Images