Data processing method, data processing model and training method thereof

Abstract

The present disclosure provides a data processing method, a data processing model and a training method thereof, which relate to the field of artificial intelligence, and specifically to natural language processing, deep learning technology, and large model technology. The implementation scheme is: determining input data, the input data includes multiple segmented words; determining the correlation between each segmented word in the multiple segmented words and each expert network in multiple expert networks according to a gating matrix, wherein the multiple expert networks are used to strengthen the multiple segmented words; allocating the multiple segmented words to the multiple expert networks in a uniform manner according to the correlation and the preset capacity of each expert network to strengthen the multiple segmented words; and determining the data processing result according to the strengthened multiple segmented words.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically to natural language processing, deep learning technology, and large model technology, and more specifically to a data processing method, a data processing model and a training method thereof, a data processing device, a training device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Pre-trained language models are a rapidly developing technology in the field of natural language processing in recent years. In large-scale model technology, generally speaking, by increasing the size of the pre-trained model, the model can achieve better results.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a data processing method, a data processing model and a training method thereof, a data processing device, a training device, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, there is provided a data processing method, comprising: determining input data, the input data comprising a plurality of segmentations; determining, according to a gating matrix, a correlation between each segmentation in the plurality of segmentations and each expert network in a plurality of expert networks, wherein the plurality of expert networks are used to strengthen the plurality of segmentations; distributing the plurality of segmentations to the plurality of expert networks in a uniform manner according to the correlation and a preset capacity of each expert network, so as to strengthen the plurality of segmentations; and determining a data processing result according to the strengthened plurality of segmentations.

[0007] According to another aspect of the present disclosure, there is provided a data processing model, comprising: a routing layer, configured to determine, according to a gating matrix, a correlation between each of a plurality of word segmentations in input data and each of a plurality of expert networks, wherein the correlation is used to distribute the plurality of word segmentations to the plurality of expert networks in a uniform manner in combination with a preset capacity of each expert network; an expert network layer comprising a plurality of expert networks, the expert networks being configured to strengthen the distributed word segmentations; and an output layer, configured to determine a data processing result according to the strengthened plurality of word segmentations.

[0008] According to another aspect of the present disclosure, a method for training a data processing model is provided, comprising: determining a sample input, the sample input comprising a plurality of segmentations; determining a correlation between each of the plurality of segmentations and each of a plurality of expert networks according to a gating matrix, wherein the plurality of expert networks are used to strengthen the plurality of segmentations, and the correlation is used to distribute the plurality of segmentations to the plurality of expert networks in a uniform manner in combination with a preset capacity of each expert network; determining a loss value based on the gating matrix, wherein the closer the gating matrix is ​​to an orthogonal matrix, the smaller the loss value; and adjusting the parameters of the gating matrix based on the loss value to obtain a trained data processing model.

[0009] According to another aspect of the present disclosure, there is provided a data processing device, comprising: an input unit, configured to determine input data, the input data comprising a plurality of segmentations; a correlation determination unit, configured to determine the correlation between each of the plurality of segmentations and each of a plurality of expert networks according to a gating matrix, wherein the plurality of expert networks are used to strengthen the plurality of segmentations; an allocation unit, configured to allocate the plurality of segmentations to the plurality of expert networks in a uniform manner according to the correlation and a preset capacity of each expert network, so as to strengthen the plurality of segmentations; and an output unit, configured to determine a data processing result according to the strengthened plurality of segmentations.

[0010] According to another aspect of the present disclosure, there is provided an apparatus for training a data processing model, comprising: an input unit configured to determine a sample input, wherein the sample input comprises a plurality of segmentations; a correlation determination unit configured to determine a correlation between each of the plurality of segmentations and each of a plurality of expert networks according to a gating matrix, wherein the plurality of expert networks are used to strengthen the plurality of segmentations, and the correlation is used to distribute the plurality of segmentations to the plurality of expert networks in a uniform manner in combination with a preset capacity of each expert network; a loss determination unit configured to determine a loss value based on the gating matrix, wherein the closer the gating matrix is ​​to an orthogonal matrix, the smaller the loss value; and a parameter adjustment unit configured to adjust a parameter of the gating matrix based on the loss value to obtain a trained data processing model.

[0011] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the present disclosure.

[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method provided by the present disclosure.

[0013] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method provided by the present disclosure when executed by a processor.

[0014] According to one or more embodiments of the present disclosure, when allocating data processing tasks to multiple expert networks, optimal allocation can be considered while satisfying the constraint of uniform allocation, so that each expert network can play a role in the data processing process and improve data processing efficiency.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0018] Figure 2 An exemplary process of a data processing method according to an embodiment of the present disclosure is shown;

[0019] Figure 3 An exemplary block diagram of a data processing model according to an embodiment of the present disclosure is shown;

[0020] Figure 4 An exemplary flow chart of a method for training a data processing model according to an embodiment of the present disclosure is shown;

[0021] Figure 5A An example of a data processing process according to the related art is shown;

[0022] Figure 5B An example of a data processing process according to an embodiment of the present disclosure is shown;

[0023] Figure 6 An exemplary block diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0024] Figure 7 An exemplary block diagram of an apparatus for training a data processing model according to an embodiment of the present disclosure is shown;

[0025] Figure 8 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0028] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0029] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0030] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0031] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the method according to an embodiment of the present disclosure.

[0032] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0033] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0034] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to obtain data processed by the method according to the embodiments of the present disclosure. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0035] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0036] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0037] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0038] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0039] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0040] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0041] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0042] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0043] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0044] Figure 2 An exemplary process of a data processing method according to an embodiment of the present disclosure is shown.

[0045] like Figure 2 As shown, in step S202, input data may be determined, and the input data may include a plurality of segmented words.

[0046] In step S204, the correlation between each of the multiple word segments and each of the multiple expert networks may be determined according to the gating matrix, wherein the multiple expert networks are used to strengthen the multiple word segments.

[0047] In step S206, the multiple segmented words may be evenly distributed to the multiple expert networks according to the correlation and the preset capacity of each expert network, so as to strengthen the multiple segmented words.

[0048] In step S208, a data processing result may be determined based on the multiple enhanced word segments.

[0049] By utilizing the embodiments of the present disclosure, when allocating data processing tasks to multiple expert networks, optimal allocation can be considered while satisfying the constraint of uniform allocation, so that each expert network can play a role in the data processing process and improve data processing efficiency.

[0050] The principles of the present disclosure will be described in detail below.

[0051] The Transformer model is a neural network model based on the attention mechanism, which is mainly used for natural language processing (NLP) tasks such as language translation and text classification. The Transformer structure is widely used in the field of natural language processing. The Transformer model mainly consists of two parts: the encoder and the decoder. The encoder converts the input sequence into a vector representation of a fixed length, while the decoder converts the vector into an output sequence. Between the encoder and the decoder, there are one or more attention mechanism layers to capture the correlation between the input sequence and the output sequence. Specifically, the Transformer model mainly consists of multiple layers of TransformerLayer, each of which contains two main parts: the self-attention layer (Self-Attention) and the feed-forward network (FFN). The self-attention mechanism is a method that calculates the correlation between each word and other words and combines them. This method can capture the relationship between each element in the input sequence, regardless of their distance in the sequence. In the Transformer model, the self-attention mechanism is used to calculate the influence of each element in the input sequence or output sequence on other elements. The feed-forward neural network is a neural network based on a fully connected layer, which is used to perform nonlinear transformations on the input sequence. In the Transformer model, FFN can be composed of two fully connected layers.

[0052] The current Transformer network structure generally uses a large number of matrix multiplications, which makes the size of the network parameters proportional to the number of matrix operations. This leads to the following series of problems in practical applications:

[0053] 1. High computational complexity. The Transformer-based model has a complex structure and contains a large number of matrix multiplications and attention mechanism calculations, which results in a long computation and training time. This becomes an important bottleneck for large-scale datasets and real-time applications.

[0054] 2. High resource consumption. Due to the high computational complexity of Transformer-based models, their training and inference processes require a lot of computing resources, including GPUs, TPUs, etc. This is a challenge for environments with limited resources.

[0055] 3. Limited ability to process long sequences. Transformer-based models perform well when processing short sequence data, but may encounter performance degradation when processing long sequence data. This is mainly because the model's self-attention mechanism needs to calculate a large number of attention scores when processing long sequences, resulting in reduced computational efficiency.

[0056] It can be seen that the Transformer network structure requires a large number of parameters to achieve the best effect. Increasing the number of model parameters exponentially means increasing computing resources exponentially for traditional structures.

[0057] The data processing model disclosed in the present invention involves a Mix of Experts (MoE) in a large-scale pre-trained model (such as a large model, a large language model). Simply put, the MoE model increases the number of feed-forward network (FFN) modules in the Transformer structure, with each feed-forward network acting as an expert. During the data processing process, different word segmentations (also called tokens or symbols, token) of the input model will be selectively distributed to different experts for calculation to achieve the strengthening of the word segmentations (for example, feature vectors representing the word segmentations). In this way, the number of parameters of the model can be expanded (for example, when Z experts are set, the parameters of the FFN module can be expanded by Z times), while the amount of computation is the same as before.

[0058] The advantage of the hybrid expert network is that it can achieve the decoupling of computing and storage, which means that computing resources and storage capacity can be independently expanded and optimized. This feature enables the hybrid expert network provided by the embodiment of the present disclosure to accommodate more model parameters within limited computing resources, thereby improving the expressiveness and performance of the model. This decoupling design can also flexibly adjust the allocation of computing and storage resources according to actual needs to achieve optimal performance and efficiency. Therefore, the hybrid expert network provided by the embodiment of the present disclosure provides an effective solution for the efficient use of computing resources, especially in a resource-constrained environment, which can bring greater advantages.

[0059] In step S202, input data may be determined, where the input data includes a plurality of segmented words.

[0060] In the example of text processing, a text segmentation algorithm can be used to process the natural language input to obtain corresponding multiple segmentations. The segmentations in the input data may include tokens corresponding to single characters or words in the input text, and may also include tokens corresponding to special symbols. Special symbols may include, for example, a sentence start symbol [CLS], a symbol used to separate different sentences or at the end of a sentence [SEP], and so on.

[0061] When using the Transformer-based network structure in other data processing fields, word segmentation can correspondingly represent data fragments obtained by segmenting other data (such as images, etc.).

[0062] In step S204, the correlation between each of the multiple word segments and each of the multiple expert networks may be determined according to the gating matrix, wherein the multiple expert networks are used to strengthen the multiple word segments.

[0063] In the MoE model, the routing module is used to distribute the input word segmentation to the corresponding expert network. The routing module is equivalent to a projection, which projects the vector expression of the input word segmentation to each expert network. The routing module can be expressed as formula (1):

[0064] gate=softmax(Linear(X)) (1)

[0065] Among them, X represents input data, Linear represents the function of linear mapping, and softmax represents normalizing the result of linear mapping. The result output by the routing module can be used to represent the correlation between each word segmentation and each expert network in multiple expert networks.

[0066] For example, a routing module can be composed of a gate matrix Implementation, d is the dimension of the word segmentation (for example, the feature vector representing the word segmentation), and e is the number of expert networks. The vectors in the gating matrix represent the learnable parameters of each expert network. Using the gating matrix, the correlation between each word segmentation and each expert network can be calculated. In some examples, the correlation between each expert network in multiple expert networks and each word segmentation in multiple word segmentations can be determined based on the product of the input data (including the vector representation of multiple word segmentations) and the gating matrix. The gating matrix can be used to conveniently determine the correlation between each input word segmentation and each expert network. In the example of the gating matrix, the routing module can be expressed as formula (2):

[0067] gate=softmax(W·X) (2)

[0068] Where X represents the input data and W represents the gating matrix.

[0069] In step S206, the multiple segmented words may be evenly distributed to the multiple expert networks according to the correlation and the preset capacity of each expert network, so as to strengthen the multiple segmented words.

[0070] In the related art, during each forward process of the network, the word segmentation is processed according to the expert network with the greatest relevance to the word segmentation determined in the routing module. However, based on such an allocation method, a phenomenon of "model collapse" will occur, which is manifested in that for multiple newly introduced experts, the model tends to use only 1-2 of the "experts", resulting in the remaining "expert" networks being unable to be effectively trained and used, thereby failing to effectively improve the prediction effect of the network. In the related art, an auxiliary loss function is used to guide so that each "expert" network has the possibility of being trained. The auxiliary loss function usually has the form of the following formula (3):

[0071] L aux =Entropy(gate) (3)

[0072] The prediction results of the routing module are guided to be averaged by increasing the entropy of the gated part used for prediction. However, even when the auxiliary loss function is used for guidance, in actual network operation, multiple tokens need to be scored and predicted in parallel to ensure high computing performance. When several tokens are "routed" in parallel, "clustering" will occur, which is manifested as inconsistent numbers of tokens obtained by each "expert". For contemporary computing hardware, this inconsistency will lead to low computing efficiency.

[0073] In order to solve the above problems, the present disclosure adopts a uniform distribution method to distribute multiple word segments to different expert networks, so that all expert networks use computing resources in a uniform manner to improve computing efficiency.

[0074] The predetermined capacity of each expert network, i.e., the number of segmented words that each expert network can process, can be determined based on the number of segmented words included in the input data and the number of expert networks. For example, the predetermined capacity of each expert network can be determined as M / e, where M is the number of segmented words in the input data and e is the number of expert networks. In some examples, the result of M / e can be rounded up or down to determine the predetermined capacity of each expert network. Each time a segmented word is assigned to an expert network, a portion of the capacity of the expert network is occupied.

[0075] In some embodiments, for each segmentation, the expert network with the highest correlation with the segmentation can be determined using the correlation determined according to the gating matrix. If the remaining capacity of the expert network with the highest correlation is not zero, the segmentation can be assigned to the expert network with remaining capacity. The correlation of multiple segmentations can be calculated in sequence and assigned to the expert networks.

[0076] In other embodiments, multiple allocations can be performed on multiple word segmentations. Each allocation will cause the expert network to strengthen multiple input word segmentations once. The amount of computation of the network for the input word segmentations can be adjusted by setting the number of allocations. It can be seen that the computing resources required to be occupied by the data processing model provided according to the embodiment of the present disclosure are determined based on the number of input word segmentations and the number of times the expert network processes the word segmentations, while the amount of parameters to be stored in the data processing model is determined based on the number of expert networks. There is no correlation between the two. Therefore, the decoupling of computing and storage can be achieved.

[0077] In the case of multiple allocations, each word segment can be allocated to the expert network with the highest relevance to the word segment in one of the allocation schemes, and constraints can be added in another allocation scheme to ensure that the final allocation result is a uniform distribution.

[0078] For example, multiple segmentations may be allocated twice. In the first allocation, each segmentation may be allocated to the expert network with the highest relevance to the segmentation. In the second allocation, each segmentation is allocated to the expert network with the highest relevance as much as possible under the constraint of uniform allocation. The result of the two allocations is that the number of segmentations processed by each expert network is basically the same, so that uniform allocation of computing resources can be achieved. In some examples, uniform allocation can be achieved based on an optimal transport algorithm.

[0079] An optimal transport algorithm is an algorithm used in mathematics and optimization to solve transportation problems. In its most basic form, it involves how to move items from a set of locations to another set of locations with the least cost. This problem has many variations in real-world applications, including logistics, image processing, machine learning, and other fields. In an optimal transport algorithm, there is usually a "cost matrix" whose elements represent the cost of moving from one location to another. The goal of the algorithm is to find a transportation solution that satisfies all demand and supply constraints while minimizing the total transportation cost. To solve this problem, optimal transport algorithms usually combine mathematical tools such as linear programming and graph theory. For large-scale problems, more efficient computational techniques and optimization methods are also needed. In the context of machine learning and deep learning, optimal transport algorithms have been used as a tool to improve the performance of models, especially when it comes to alignment, matching, or transfer learning between data distributions. It can help establish better correspondence between different data sets, thereby improving the generalization ability of the model. Using optimal transport algorithms, efficient and low-cost resource allocation and transportation solutions can be achieved in various application scenarios.

[0080] In the embodiments of the present disclosure, the problem of allocating word segments can be converted into a transportation problem, so that the optimal transportation algorithm can be used to evenly allocate word segments to various expert networks.

[0081] In some examples, step S206 may include: determining a first allocation scheme according to the correlation, in which each segmentation is allocated to the expert network with the highest correlation with the segmentation. Determine the remaining capacity of each expert network according to the first allocation scheme and the preset capacity of each expert network. Determine a second allocation scheme according to the correlation and the remaining capacity of each expert network, wherein the second allocation scheme is combined with the first allocation scheme so that multiple segmentations are allocated to multiple expert networks in a uniform manner. For example, the second allocation scheme may be determined based on the optimal transport algorithm according to the correlation and the remaining capacity of each expert network. In the example, the optimal transport algorithm may be a sinkhorn algorithm or any other suitable optimal transport algorithm. The sinkhorn algorithm may satisfy the constraint of uniform distribution while considering the cost (correlation between the segmentation and the expert network). The sinkhorn algorithm may be efficiently implemented on acceleration hardware such as a GPU. By converting the process of allocating segmentations to expert networks into an optimal transport problem, using the optimal transport algorithm, each segmentation may be allowed to select the expert with the highest correlation with itself as much as possible while ensuring the uniformity of the transport results, so that each expert processes the same (or substantially the same) number of segmentations, achieving optimal computational efficiency.

[0082] In some examples, the preset capacity of each expert network may be the same. In the case of multiple allocations, the preset capacity of each expert network may be determined based on (N×M) / e, where N represents the number of allocations, M is the number of word segments in the input data, and e is the number of expert networks.

[0083] In step S208, the data processing result can be determined based on the multiple reinforced word segments. For example, the expert network implements the function of the FFN in the Transformer network structure. After the assigned word segments are reinforced via the expert network, the reinforced word segments can be input into the next layer in the network, such as the self-attention layer. In some examples, step S208 can include processing the reinforced multiple word segments based on the self-attention mechanism to obtain the data processing result.

[0084] Figure 3 An exemplary block diagram of a data processing model according to an embodiment of the present disclosure is shown. Figure 3 The data processing model shown in Figure 2 Combined with the data processing method described in Figure 3 The data processing model described may be a deep neural network or any other mathematical model consistent with the principles of the present disclosure. It may be used in large models (such as large language models) Figure 3 The structure of the data processing model is shown in .

[0085] like Figure 3 As shown, the data processing model may include a routing layer 310 , an expert network layer 320 including a plurality of expert networks, and an output layer 330 .

[0086] The routing layer 310 may be configured to determine a correlation between each of the multiple word segmentations in the input data and each of the multiple expert networks according to the gating matrix, wherein the correlation is used to distribute the multiple word segmentations to the multiple expert networks in a uniform manner in combination with a preset capacity of each expert network.

[0087] The expert network in the expert network layer 320 may be configured to enhance the assigned word segments.

[0088] The output layer 330 may be configured to determine a data processing result according to the strengthened plurality of word segmentations.

[0089] In some embodiments, determining the correlation between each of the multiple word segmentations and each of the multiple expert networks according to the gating matrix may include: determining the correlation between each of the multiple expert networks and each of the multiple word segmentations based on the product of the input data and the gating matrix.

[0090] In some embodiments, allocating multiple segmentations to multiple expert networks in a uniform manner may include: determining a first allocation scheme based on relevance, in which each segmentation is allocated to the expert network with the highest relevance to the segmentation; determining the remaining capacity of each expert network based on the first allocation scheme and the preset capacity of each expert network; and determining a second allocation scheme based on the relevance and the remaining capacity of each expert network, wherein the second allocation scheme is combined with the first allocation scheme so that the multiple segmentations are allocated to the multiple expert networks in a uniform manner.

[0091] In some embodiments, the preset capacity of each expert network may be the same.

[0092] In some embodiments, determining the second allocation scheme according to the correlation and the remaining capacity of each expert network may include: determining the second allocation scheme according to the correlation and the remaining capacity of each expert network based on an optimal transport algorithm.

[0093] In some embodiments, the optimal transport algorithm may be a sinkhorn algorithm.

[0094] In some embodiments, the output layer can be configured to: process the reinforced multiple word segmentations based on a self-attention mechanism to obtain a data processing result.

[0095] Figure 4An exemplary flow chart of a method for training a data processing model according to an embodiment of the present disclosure is shown. Figure 4 The method shown in the Figure 3 The described data processing model is trained.

[0096] In step S402, a sample input may be determined, wherein the sample input includes a plurality of segmented words. The content of the sample input may be selected from a predetermined training data set.

[0097] In step S404, the correlation between each of the input multiple word segments and each of the multiple expert networks can be determined according to the gating matrix. The multiple expert networks are used to strengthen the multiple word segments. The correlation determined in step S204 can be used to distribute the multiple word segments to the multiple expert networks in a uniform manner in combination with the preset capacity of each expert network. The preset capacity of each expert network can be the same, and the preset capacity of each expert network can be used in combination with the present disclosure. Figure 2 The described method determines a preset capacity for each expert network.

[0098] In step S406, a loss value may be determined based on the gating matrix. The closer the gating matrix is ​​to the orthogonal matrix, the smaller the loss value. In some embodiments, the loss value may be designed based on the characteristics of the orthogonal matrix. One of the characteristics of the orthogonal matrix is ​​that the product of the orthogonal matrix and its transposed matrix is ​​the unit matrix. Therefore, the product of the gating matrix and its transposed matrix may be determined, and the loss value may be determined based on the error between the gating matrix and the unit matrix. For example, the loss value may be determined based on the following formula (4):

[0099] L orth =MSE(W·W T ,I) (4)

[0100] Among them, L orth represents the loss value, MSE represents the minimum mean square error, W represents the gating matrix, and W T represents the transposed matrix of the gating matrix, and I represents the identity matrix. It can be understood that although the MSE is used as an example in formula (4) to describe the calculation method of the loss value, other forms of statistical errors can also be used to determine the loss value without departing from the principles of the present disclosure.

[0101] In step S408, the parameters of the gating matrix may be adjusted based on the loss value determined in step S406 to obtain a trained data processing model.

[0102] By using the model training method provided by the embodiments of the present disclosure, the loss value can be set so that the vectors corresponding to each expert network in the gate matrix used to represent the trainable parameters of each expert network in the trained data processing model are mutually (close to) orthogonal, so that each expert network can strengthen the word segmentation in a different way. When the gate matrix is ​​close to an orthogonal matrix, each expert network is related to each word segmentation in a different way, so that each expert network tends to select a different word segmentation, avoiding the situation where the expert network tends to select the same word segmentation when assigning an expert network to each word segmentation, resulting in the omission of some word segmentations.

[0103] Figure 5A An example of a data processing process according to the related art is shown. Figure 5A In the example shown, the parameters of the routing module are homogenized. Specifically, the dot product angles of the vectors of each expert network in the gating matrix are close to each other, as shown in formula (5):

[0104]

[0105] Among them, W i , W j They represent the vectors for the i-th and j-th expert networks in the gating matrix respectively, and D represents the index set of the expert network.

[0106] exist Figure 5A In the example shown, due to the homogenization phenomenon, different expert networks 510 and 520 tend to select the same segmented words (such as C, D, E, F, G) for processing, resulting in some segmented words (A, B) being omitted and unable to be processed.

[0107] Figure 5B An example of a data processing process according to an embodiment of the present disclosure is shown. Figure 4 The vectors of each expert network in the gating matrix of the data processing model obtained by training with the training method described in the previous section tend to be orthogonal, so different expert networks can choose different word segmentations for processing, avoiding Figure 5A The homogeneity problem that arises in .

[0108] Figure 6 6 shows an exemplary block diagram of a data processing device according to an embodiment of the present disclosure. As shown in FIG6 , the data processing device 600 may include an input unit 610, a correlation determination unit 620, an allocation unit 630, and an output unit 640. Figure 6 The data processing device described in the embodiment performs the combination Figure 2 Describe the data processing methods.

[0109] The input unit 610 may be configured to determine input data, the input data including a plurality of word segments.

[0110] The correlation determination unit 620 may be configured to determine the correlation between each of the multiple word segmentations and each of the multiple expert networks according to the gating matrix, wherein the multiple expert networks are used to strengthen the multiple word segmentations.

[0111] The allocating unit 630 may be configured to allocate the plurality of segmented words to the plurality of expert networks in a uniform manner according to the relevance and the preset capacity of each expert network, so as to strengthen the plurality of segmented words.

[0112] The output unit 640 may be configured to determine a data processing result according to the enhanced plurality of word segmentations.

[0113] In some embodiments, the relevance determination unit is configured to include: determining the relevance of each of the plurality of expert networks and each of the plurality of word segments based on a product of the input data and the gating matrix.

[0114] In some embodiments, allocating multiple segmentations to multiple expert networks in a uniform manner based on relevance and a preset capacity of each expert network may include: determining a first allocation scheme based on relevance, in which each segmentation is allocated to an expert network having the highest relevance to the segmentation; determining the remaining capacity of each expert network based on the first allocation scheme and the preset capacity of each expert network; and determining a second allocation scheme based on the relevance and the remaining capacity of each expert network, wherein the second allocation scheme is combined with the first allocation scheme so that multiple segmentations are allocated to multiple expert networks in a uniform manner.

[0115] In some embodiments, the preset capacity of each expert network may be the same.

[0116] In some embodiments, determining the second allocation scheme according to the correlation and the remaining capacity of each expert network may include: determining the second allocation scheme according to the correlation and the remaining capacity of each expert network based on an optimal transport algorithm.

[0117] In some embodiments, the optimal transport algorithm may be a sinkhorn algorithm.

[0118] In some embodiments, determining a data processing result based on the multiple enhanced word segmentations may include: processing the multiple enhanced word segmentations based on a self-attention mechanism to obtain a data processing result.

[0119] It should be understood that Figure 6 The various modules or units of the apparatus 600 shown in FIG. 6 can be used in conjunction with the reference Figure 2The steps in the method 200 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 600 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0120] Figure 7 FIG. 7 shows an exemplary block diagram of an apparatus for training a data processing model according to an embodiment of the present disclosure. As shown in FIG. 7 , the apparatus 700 may include an input unit 710, a correlation determination unit 720, a loss determination unit 730, and a parameter adjustment unit 740. Figure 7 The data processing device described in the embodiment performs the combination Figure 4 A method for training a data processing model is described.

[0121] The input unit 710 may be configured to determine a sample input, where the sample input includes a plurality of word segments.

[0122] The correlation determination unit 720 may be configured to determine the correlation between each of the multiple segmented words and each of the multiple expert networks according to the gating matrix, wherein the multiple expert networks are used to strengthen the multiple segmented words, and the correlation is used to distribute the multiple segmented words to the multiple expert networks in a uniform manner in combination with the preset capacity of each expert network.

[0123] The loss determination unit 730 may be configured to determine a loss value based on a gating matrix, wherein the closer the gating matrix is ​​to an orthogonal matrix, the smaller the loss value.

[0124] The parameter adjustment unit 740 may be configured to adjust the parameters of the gating matrix based on the loss value to obtain a trained data processing model.

[0125] In some embodiments, the loss determination unit may be configured to: determine a product of a gating matrix and a transposed matrix of the gating matrix; and determine the loss value based on an error between the product and an identity matrix.

[0126] In some embodiments, the error is a minimum mean square error.

[0127] It should be understood that Figure 7 The various modules or units of the apparatus 700 shown in FIG. 7 can be used in conjunction with the reference Figure 4 The steps in the method 400 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 400 are also applicable to the device 700 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0128] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various units discussed herein may be separated into multiple units, and / or at least some functionality of multiple units may be combined into a single unit.

[0129] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 6 , Figure 7 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 610 to 640, units 710 to 740 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0130] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method according to an embodiment of the present disclosure.

[0131] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, where the computer instructions are used to cause the computer to execute the method according to the embodiment of the present disclosure.

[0132] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the method according to the embodiment of the present disclosure when being executed by a processor.

[0133] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0134] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0135] refer to Figure 8 , a block diagram of an electronic device 800 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0136] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0137] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808 and a communication unit 809. The input unit 806 can be any type of device that can input information to the electronic device 800. The input unit 806 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone and / or a remote control. The output unit 807 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 808 can include but is not limited to a disk, an optical disk. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth TMdevices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0138] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as methods 200 and 400. For example, in some embodiments, methods 200 and 400 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the methods 200 and 400 described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the methods 200 , 400 in any other appropriate manner (eg, by means of firmware).

[0139] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0141] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0143] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0144] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0145] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0146] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A data processing method, comprising: Determine input data, wherein the input data includes a plurality of segmented words; Determine, according to a gating matrix, a correlation between each of the plurality of segmentations and each of the plurality of expert networks, wherein the plurality of expert networks are used to strengthen the plurality of segmentations; Allocating the plurality of segmented words to the plurality of expert networks in a uniform manner according to the correlation and a preset capacity of each expert network to strengthen the plurality of segmented words; Determine a data processing result according to the multiple enhanced word segments, The step of uniformly distributing the plurality of segmented words to the plurality of expert networks according to the correlation and the preset capacity of each expert network includes: Determining a first allocation scheme according to the correlation, in which each word segment is allocated to an expert network having the highest correlation with the word segment; Determining the remaining capacity of each expert network according to the first allocation scheme and the preset capacity of each expert network; A second allocation scheme is determined according to the correlation and the remaining capacity of each expert network, wherein the second allocation scheme is combined with the first allocation scheme so that the plurality of segmentations are evenly allocated to the plurality of expert networks.

2. The data processing method according to claim 1, wherein: Determining the correlation between each of the multiple word segments and each of the multiple expert networks according to the gating matrix includes: Based on the product of the input data and the gating matrix, the correlation between each of the plurality of expert networks and each of the plurality of word segments is determined.

3. The data processing method according to claim 1, wherein: The preset capacity of each expert network is the same.

4. The data processing method according to claim 1, wherein: Determining a second allocation scheme according to the correlation and the remaining capacity of each expert network includes: Based on the optimal transport algorithm, a second allocation scheme is determined according to the correlation and the remaining capacity of each expert network.

5. The data processing method according to claim 4, wherein: The optimal transport algorithm is the sinkhorn algorithm.

6. The data processing method according to claim 1, wherein: Determining data processing results according to the multiple enhanced word segmentations includes: The reinforced multiple word segmentations are processed based on a self-attention mechanism to obtain the data processing result.

7. A data processing method implemented by a data processing model, wherein: The data processing model includes a routing layer, an expert network layer including a plurality of expert networks, and an output layer, and the method includes: The routing layer determines, according to the gating matrix, a correlation between each of a plurality of segmented words in the input data and each of a plurality of expert networks, wherein the correlation is used to distribute the plurality of segmented words to the plurality of expert networks in a uniform manner in combination with a preset capacity of each expert network; The expert network in the expert network layer strengthens the assigned word segmentation; The output layer determines a data processing result according to the multiple enhanced word segmentations, The step of uniformly distributing the plurality of segmented words to the plurality of expert networks comprises: Determining a first allocation scheme according to the correlation, in which each word segment is allocated to an expert network having the highest correlation with the word segment; Determining the remaining capacity of each expert network according to the first allocation scheme and the preset capacity of each expert network; A second allocation scheme is determined according to the correlation and the remaining capacity of each expert network, wherein the second allocation scheme is combined with the first allocation scheme so that the plurality of segmentations are evenly allocated to the plurality of expert networks.

8. The data processing method according to claim 7, wherein: Determining the correlation between each of the multiple word segments and each of the multiple expert networks according to the gating matrix includes: Based on the product of the input data and the gating matrix, the correlation between each of the plurality of expert networks and each of the plurality of word segments is determined.

9. The data processing method according to claim 7, wherein: The preset capacity of each expert network is the same.

10. The data processing method according to claim 7, wherein: Determining a second allocation scheme according to the correlation and the remaining capacity of each expert network includes: Based on the optimal transport algorithm, a second allocation scheme is determined according to the correlation and the remaining capacity of each expert network.

11. The data processing method according to claim 10, wherein: The optimal transport algorithm is the sinkhorn algorithm.

12. The data processing method according to claim 7, wherein: Determining data processing results according to the multiple enhanced word segmentations includes: The reinforced multiple word segmentations are processed based on a self-attention mechanism to obtain the data processing result.

13. A method for training a data processing model, comprising: Determine a sample input, wherein the sample input includes a plurality of segmented words; Determining, according to a gating matrix, a correlation between each of the plurality of segmented words and each of the plurality of expert networks, wherein the plurality of expert networks are used to strengthen the plurality of segmented words, and the correlation is used to distribute the plurality of segmented words to the plurality of expert networks in a uniform manner in combination with a preset capacity of each expert network; Determine a loss value based on the gating matrix, wherein the closer the gating matrix is ​​to an orthogonal matrix, the smaller the loss value; adjusting the parameters of the gating matrix based on the loss value to obtain a trained data processing model, The step of uniformly distributing the plurality of segmented words to the plurality of expert networks according to the correlation and the preset capacity of each expert network includes: Determining a first allocation scheme according to the correlation, in which each word segment is allocated to an expert network having the highest correlation with the word segment; Determining the remaining capacity of each expert network according to the first allocation scheme and the preset capacity of each expert network; A second allocation scheme is determined according to the correlation and the remaining capacity of each expert network, wherein the second allocation scheme is combined with the first allocation scheme so that the plurality of segmentations are evenly allocated to the plurality of expert networks.

14. The method of claim 13, wherein: Determining the loss value based on the gating matrix includes: determining a product of the gating matrix and a transposed matrix of the gating matrix; The loss value is determined based on an error between the product and an identity matrix.

15. The method of claim 14, wherein: The error is the minimum mean square error.

16. A data processing device, comprising: An input unit configured to determine input data, wherein the input data includes a plurality of segmented words; a correlation determination unit configured to determine, according to a gating matrix, a correlation between each of the plurality of segmented words and each of the plurality of expert networks, wherein the plurality of expert networks are used to strengthen the plurality of segmented words; an allocating unit configured to allocate the plurality of segmented words to the plurality of expert networks in a uniform manner according to the correlation and a preset capacity of each expert network, so as to strengthen the plurality of segmented words; an output unit configured to determine a data processing result according to the plurality of enhanced word segments, The step of uniformly distributing the plurality of segmented words to the plurality of expert networks according to the correlation and the preset capacity of each expert network includes: Determining a first allocation scheme according to the correlation, in which each word segment is allocated to an expert network having the highest correlation with the word segment; Determining the remaining capacity of each expert network according to the first allocation scheme and the preset capacity of each expert network; A second allocation scheme is determined according to the correlation and the remaining capacity of each expert network, wherein the second allocation scheme is combined with the first allocation scheme so that the plurality of segmentations are evenly allocated to the plurality of expert networks.

17. The data processing apparatus according to claim 16, wherein: The correlation determination unit is configured to include: Based on the product of the input data and the gating matrix, the correlation between each of the plurality of expert networks and each of the plurality of word segments is determined.

18. The data processing apparatus according to claim 16, wherein: The preset capacity of each expert network is the same.

19. The data processing apparatus according to claim 16, wherein: Determining a second allocation scheme according to the correlation and the remaining capacity of each expert network includes: Based on the optimal transport algorithm, a second allocation scheme is determined according to the correlation and the remaining capacity of each expert network.

20. The data processing apparatus according to claim 19, wherein: The optimal transport algorithm is the sinkhorn algorithm.

21. The data processing apparatus according to claim 16, wherein: Determining data processing results according to the multiple enhanced word segmentations includes: The reinforced multiple word segmentations are processed based on a self-attention mechanism to obtain the data processing result.

22. An apparatus for training a data processing model, comprising: An input unit, configured to determine a sample input, wherein the sample input includes a plurality of segmented words; a correlation determination unit configured to determine, according to a gating matrix, a correlation between each of the plurality of segmented words and each of the plurality of expert networks, wherein the plurality of expert networks are used to strengthen the plurality of segmented words, and the correlation is used to distribute the plurality of segmented words to the plurality of expert networks in a uniform manner in combination with a preset capacity of each expert network; a loss determination unit configured to determine a loss value based on the gating matrix, wherein the closer the gating matrix is ​​to an orthogonal matrix, the smaller the loss value; a parameter adjustment unit, configured to adjust the parameters of the gating matrix based on the loss value to obtain a trained data processing model, The step of uniformly distributing the plurality of segmented words to the plurality of expert networks according to the correlation and the preset capacity of each expert network includes: Determining a first allocation scheme according to the correlation, in which each word segment is allocated to an expert network having the highest correlation with the word segment; Determining the remaining capacity of each expert network according to the first allocation scheme and the preset capacity of each expert network; A second allocation scheme is determined according to the correlation and the remaining capacity of each expert network, wherein the second allocation scheme is combined with the first allocation scheme so that the plurality of segmentations are evenly allocated to the plurality of expert networks.

23. The device of claim 22, wherein: The loss determination unit is configured to: determining a product of the gating matrix and a transposed matrix of the gating matrix; The loss value is determined based on an error between the product and an identity matrix.

24. The device of claim 23, wherein: The error is the minimum mean square error.

25. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-15.

27. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images