Data processing method and device, computer equipment, storage medium and program product

Abstract

The embodiment of the application provides a kind of data processing method, device and computer equipment, storage medium, program product, which comprises: calling business processing model to carry out business processing to sample data, expert submodel in business processing model includes multiple expert networks;In the process of business processing, the expert allocation information of each sample unit in sample data in expert submodel is obtained;Based on the expert allocation information of each sample unit in expert submodel, the expert load balancing loss of business processing model is determined;Expert load balancing loss is used to represent: the load balancing degree of each expert network in expert submodel;According to the rule of reducing the expert load balancing loss of business processing model, business processing model is trained.Using the embodiment of the application, the load imbalance problem of each expert network in expert submodel (i.e. MoE) can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of artificial intelligence technology, specifically to a data processing method, a data processing device, a computer equipment, a computer-readable storage medium, and a computer program product. Background Technology

[0002] As research into artificial intelligence technology deepens, researchers have discovered that the more training data a model has and the larger the scale of its parameters, the better its training performance. For example, in the field of natural language processing (NLP) within AI technology, pre-trained language models (PTMs) have elevated NLP research and applications to a new level, achieving significant results in numerous NLP tasks such as sentiment analysis, named entity recognition, and question answering. The GPT-2 model (a pre-trained language processing model) has 1.5 billion parameters, the Google T5 model (a pre-trained language processing model) has 11 billion parameters, and the GPT-3 model (a pre-trained language processing model) has 175 billion parameters.

[0003] However, the larger the model's parameters, the higher the computational cost required to train the model. To increase the model size without significantly increasing computational costs, Mixture-of-Experts (MoE) has been introduced. The basic idea of ​​MoE is to break down a large model into several smaller models (each of which can be called an expert network). For a given sample, several smaller models are activated for computation, thus increasing the model size without significantly increasing computational costs. Although introducing MoE into models has made some progress in reducing the computational cost of model training, the challenges in training models with MoE remain unresolved. One such challenge is the uneven load distribution among the expert networks in MoE. Therefore, how to improve the uneven load distribution among the expert networks in MoE is currently a hot research topic. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computer equipment, storage medium, and program product that can effectively improve the load imbalance problem among various expert networks in the expert sub-model (i.e., MoE).

[0005] On one hand, embodiments of this application provide a data processing method, which includes:

[0006] The business processing model is invoked to perform business processing on the sample data; the business processing model includes an expert sub-model, which includes multiple expert networks, and the sample data includes multiple sample units.

[0007] During business processing, the expert allocation information of each sample unit in the sample data in the expert sub-model is obtained; the expert allocation information includes the probability information of the sample unit being assigned to each expert network in the expert sub-model for processing.

[0008] Based on the expert allocation information of each sample unit in the expert sub-model, the expert load balancing loss of the business processing model is determined. The expert load balancing loss of the business processing model is used to characterize the load balancing degree of each expert network in the expert sub-model when each sample unit in the sample data is allocated to the expert network in the expert sub-model for processing according to the corresponding probability information.

[0009] The business processing model is trained according to the rule of reducing the expert load balancing loss of the business processing model; the trained business processing model is used to identify and understand the content expressed by the media data.

[0010] Accordingly, embodiments of this application provide a data processing apparatus, which includes:

[0011] The processing unit is used to call the business processing model to perform business processing on the sample data; the business processing model includes an expert sub-model, which includes multiple expert networks, and the sample data includes multiple sample units.

[0012] The acquisition unit is used to acquire the expert allocation information of each sample unit in the sample data in the expert sub-model during the business processing; the expert allocation information includes the probability information of the sample unit being assigned to each expert network in the expert sub-model for processing.

[0013] The processing unit is also used to determine the expert load balancing loss of the business processing model based on the expert allocation information of each sample unit in the expert sub-model. The expert load balancing loss of the business processing model is used to characterize the load balancing degree of each expert network in the expert sub-model when each sample unit in the sample data is allocated to the expert network in the expert sub-model for processing according to the corresponding probability information.

[0014] The processing unit is also used to train the business processing model according to the rules for reducing the expert load balancing loss of the business processing model; the trained business processing model is used to identify and understand the content expressed by the media data.

[0015] In one implementation, when determining the expert load balancing loss of the business processing model based on the expert allocation information of each sample unit in the expert sub-model, the processing unit specifically performs the following steps:

[0016] The information content of expert allocation information for each sample unit in the expert sub-model is calculated to obtain the information content of expert allocation for each sample unit in the expert sub-model.

[0017] Based on the difference between the amount of expert-assigned information in each sample unit in the expert sub-model and the amount of information equally assigned in the expert sub-model, the expert load balancing loss of the business processing model is determined.

[0018] The information content of equal distribution in the expert sub-model refers to the amount of information when sample units are distributed with equal probability to each expert network in the expert sub-model for processing.

[0019] In one implementation, the business processing model includes multiple expert sub-models, and the target expert sub-model is any one of the expert sub-models included in the business processing model. The processing unit, used to determine the expert load balancing loss of the business processing model based on the difference between the amount of expert-assigned information in each sample unit within the expert sub-model and the amount of information equally assigned to each expert sub-model, specifically performs the following steps:

[0020] Based on the difference between the amount of information allocated by experts to each sample unit in the target expert sub-model and the amount of information allocated equally in the target expert sub-model, calculate the information content difference of each sample unit in the target expert sub-model.

[0021] The average of the information content difference of each sample unit in the target expert sub-model is calculated to obtain the expert load balancing loss of the target expert sub-model.

[0022] The expert load balancing loss of the business processing model is obtained by summing the expert load balancing losses of each expert sub-model included in the business processing model.

[0023] In one implementation, the target sample unit is any sample unit in the sample data; the expert sub-model also includes a gating network; the acquisition unit, used during business processing to acquire the expert allocation information of each sample unit in the sample data in the expert sub-model, specifically performs the following steps:

[0024] Obtain the vector representation of the target sample unit;

[0025] By invoking the gating network and performing expert network allocation processing on the target sample unit based on its vector representation, the expert allocation information of the target sample unit in the expert sub-model is obtained.

[0026] In one implementation, the sample data includes one or more training samples, each training sample comprising multiple sample units, and the target sample unit is any sample unit in the sample data; the acquisition unit, used to acquire the vector representation of the target sample unit, specifically performs the following steps:

[0027] Obtain the intermediate vector matrix corresponding to the target training sample to which the target sample unit belongs; the intermediate vector matrix is ​​composed of the intermediate vectors of the sample units included in the target training sample;

[0028] The intermediate vector matrix is ​​subjected to attention aggregation processing to obtain the attention vector matrix; the attention vector matrix includes the attention vectors of the sample units in the target training samples;

[0029] The attention vector of the target sample unit in the attention vector matrix is ​​determined as the vector representation of the target sample unit.

[0030] In one implementation, the acquisition unit, used to perform attention aggregation processing on the intermediate vector matrix to obtain the attention vector matrix, specifically performs the following steps:

[0031] Based on the attention mechanism, h attention operations are performed on the intermediate vector matrix to obtain the attention head information corresponding to each attention operation in the h attention operations, where h is a positive integer;

[0032] The attention head information corresponding to each attention operation in h attention operations is concatenated to obtain the attention vector matrix.

[0033] In one implementation, the i-th attention operation is any one of the h attention operations, where i is a positive integer less than or equal to h; the acquisition unit, when performing h attention operations on the intermediate vector matrix based on the attention mechanism to obtain the attention head information corresponding to each of the h attention operations, specifically performs the following steps:

[0034] The intermediate vector matrix is ​​linearly transformed based on the linear transformation matrix corresponding to the i-th attention operation to obtain a linear vector matrix.

[0035] Normalize the linear vector matrix to obtain the attention head information corresponding to the i-th attention operation.

[0036] In one implementation, the acquisition unit is also used to acquire the business processing loss of the business processing model on the sample data during the business processing process.

[0037] The processing unit, when training the business processing model according to the rules for reducing the expert load balancing loss of the business processing model, specifically performs the following steps:

[0038] The loss information of the business processing model is obtained by summing the business processing loss and the expert load balancing loss of the business processing model.

[0039] The business processing model is trained according to the rules for reducing the loss information of the business processing model.

[0040] In one implementation, the business processing model includes an encoding model and a decoding model; the sample data includes input text samples and output text samples, where the input text samples include multiple input sample units and the output text samples include multiple output sample units; the acquisition unit is used to acquire the business processing loss of the business processing model on the sample data during the business processing process, specifically by performing the following steps:

[0041] The encoding model is invoked to encode each input sample unit in the input text sample, and the encoding result of each input sample unit in the input text sample is obtained.

[0042] The decoding model is invoked to combine the encoding results of each input sample unit to perform word prediction on each output sample unit in the output text sample, thereby obtaining the probability distribution information of each output sample unit in the labeled vocabulary. The probability distribution information of the output sample unit in the labeled vocabulary includes the probability that the output sample unit is predicted as each word in the labeled vocabulary.

[0043] The business processing loss is calculated based on the probability distribution information of each output sample unit in the labeled vocabulary and the labeled position vector of each output sample unit in the labeled vocabulary.

[0044] In one implementation, the encoding model includes an expert sub-model; the reference input sample unit is any input sample unit in the input text sample; the acquisition unit, used to call the encoding model to encode each input sample unit in the input text sample, specifically performs the following steps when obtaining the encoding results of each input sample unit in the input text sample:

[0045] Obtain the processing results of the expert sub-model on the reference input sample unit;

[0046] The residual standardization process is applied to the processing results of the expert sub-model on the reference input sample unit to obtain the encoding result of the reference input sample unit.

[0047] In one implementation, the acquisition unit, used to acquire the processing result of the expert sub-model on the reference input sample unit, specifically performs the following steps:

[0048] Based on the expert assignment information of the reference input sample unit in the expert sub-model, determine one or more target expert networks in the expert sub-model for computing the reference input sample unit;

[0049] The target expert network is used to perform calculations on the reference input sample units, and the calculation results of the target expert network on the reference input sample units are obtained.

[0050] Based on the probability information of the reference input sample units being assigned to the target expert network for computation, the computation results of the target expert network on the reference input sample units are weighted and summed to obtain the processing results of the expert sub-model on the reference input sample units.

[0051] In one implementation, the acquisition unit is also used to acquire raw text samples for training the business processing model, the raw text samples including multiple sample units.

[0052] The processing unit is also used to replace some sample units in the original text sample with the replacement unit to obtain the input text sample;

[0053] The processing unit is also used to concatenate the replaced sample units using the replacement unit to obtain the output text sample;

[0054] The processing unit is also used to combine the input text samples and the output text samples to obtain training samples; the sample data includes one or more training samples.

[0055] In one implementation, the acquisition unit is further configured to acquire media data to be processed, the media data including any of the following: text, images, video, and audio;

[0056] The processing unit is also used to call the trained business processing model to perform content recognition on media data and obtain the content tags corresponding to the media data.

[0057] Accordingly, embodiments of this application provide a computer device, which includes:

[0058] A processor is a tool for implementing computer programs.

[0059] A computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described data processing method.

[0060] Accordingly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned data processing method.

[0061] Accordingly, embodiments of this application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described above.

[0062] In this application embodiment, an expert load balancing loss is proposed that can be used to characterize the load balancing degree of each expert network in the expert sub-model (i.e., MoE). In other words, the expert load balancing loss is directly related to the load balancing degree of each expert network in the expert sub-model. Therefore, by training the business processing model containing the expert sub-model with the expert load balancing loss, the load imbalance problem of each expert network in the expert sub-model included in the business processing model can be effectively improved. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of the structure of an expert sub-model provided in an embodiment of this application;

[0065] Figure 2 This is a schematic diagram of the structure of an expert sub-model introduced by an embodiment of this application;

[0066] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0067] Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0068] Figure 5 This is a schematic diagram illustrating the construction process of a training sample provided in an embodiment of this application;

[0069] Figure 6 This is a schematic diagram of the structure of a business processing model provided in an embodiment of this application;

[0070] Figure 7 This is a schematic diagram of an embedding representation process of a sample unit provided in an embodiment of this application;

[0071] Figure 8This is a schematic diagram of the expert allocation process for an expert sub-model provided in an embodiment of this application;

[0072] Figure 9 This is a schematic diagram illustrating an application scenario of a business processing model provided in the real-time example of this application;

[0073] Figure 10 This is a schematic diagram of experimental results for a business processing model provided in an embodiment of this application;

[0074] Figure 11 This is a schematic diagram of the experimental results of another business processing model provided in the embodiments of this application;

[0075] Figure 12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0076] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0078] This application's embodiments involve MoE (Mixture-of-Expert, also known as expert sub-model). Here, we first introduce the definition of MoE, the methods of introducing MoE into the model, and the expert load balancing problem of MoE:

[0079] (1) Definition of MoE: MoE is a technique developed in neural networks. The basic idea of ​​MoE is to train multiple sub-neural networks (also called expert networks), where each expert network is only responsible for processing a portion of the input data. For example... Figure 1 As shown, a MoE typically includes a trainable gating network and N expert networks. These N expert networks generally have the same network structure, where N is an integer greater than 1. Given an input x, the MoE first selects several expert networks for the input through the gating network. Then, the input is assigned to the selected expert networks for computation. Finally, the results of these expert networks are aggregated to obtain the final result for the input. This aggregation can refer to a weighted average or a voting process; for example... Figure 1As shown, the selected expert networks are expert network 2 and expert network N, and G(x)2 and G(x)N represent the weights of expert network 2 and expert network N, respectively.

[0080] (2) Methods of Introducing MoE into the Model: Introducing MoE into the model typically involves extending certain sub-models or sub-layers of the model with a multi-expert network. Taking the Transformer model (a pre-trained language processing model) as an example, such as... Figure 2 As shown, the Transformer model can include a self-attention layer, an add+normalize layer, and a switching FFN layer. The FFN (Feed Forward Network) in the Transformer model can be extended as an expert network, expanding a single FFN in the FFN layer of the Transformer model into multiple FFNs, each of which acts as an expert network. At the same time, a gating network is added to control the distribution of inputs.

[0081] (3) Expert Load Balancing Problem in MoE: The expert load balancing problem refers to the tendency of the entire model to converge to a state after the introduction of MoE where a few expert networks always receive larger weights, thus processing more inputs (i.e., a few expert networks are under high load), while most expert networks receive smaller weights, thus processing very little input and almost being idle (i.e., most expert networks are under low load). This convergence state is called expert load imbalance. This expert load imbalance has a self-reinforcing characteristic: once an expert network receives a higher weight, more input will be allocated to it, making its training more thorough and further increasing its weight; conversely, an expert network with a lower weight will receive less input, leading to less thorough training and further decreasing its weight. The expert load imbalance problem will cause most expert networks in the model to not learn sufficiently, thus affecting the training effect of the entire model.

[0082] Based on the above introduction to MoE, it is clear that the expert load balancing problem in the model is closely related to the training effect; an imbalanced expert load in the MoE will lead to poor training performance. To improve the expert load imbalance problem in MoE, existing technologies have proposed a scheme to improve the expert load imbalance problem by using the importance of expert networks. The following is an introduction to the expert network importance scheme:

[0083] The expert network importance scheme improves the expert load imbalance problem in MoE by minimizing the mean squared error of expert network importance. The importance of any expert network is defined as the cumulative sum of the probabilities that each sample unit (Token) in a batch of sample data is distributed to that expert network for processing. The importance-based loss is defined as the minimum mean squared error of the importance of all expert networks. MoE is trained using the importance-based loss to encourage the importance of each expert network to be as equal as possible. Specifically:

[0084] Given a MoE model containing N expert networks, where the network number i ranges from 1 to N, and a batch of sample data is denoted as B, containing T sample units, the importance i of the i-th expert network can be described by the following formula 1:

[0085]

[0086] In Formula 1 above: p i (x) represents the probability that the gating network will assign sample unit x in the sample data to the i-th expert network for processing.

[0087] The loss based on importance can be described by the following formulas 2 and 3:

[0088]

[0089]

[0090] In formulas 2 and 3 above: L Importance This represents the loss based on importance; w Importance It is an adjustable hyperparameter used to control the weight of the loss.

[0091] As can be seen from the above description of the expert network importance scheme, the essence of this scheme is to make the importance of each expert network in MoE tend to be balanced through training. However, this scheme has the following drawback: the balance of the importance of each expert network in MoE does not guarantee the load balance of each expert network in MoE. For example, one expert network may be assigned to a small number of samples with a high probability, while another expert network may be assigned to a large number of samples with a low probability. The importance of these two expert networks is balanced, but the load is unbalanced.

[0092] Based on this, this application proposes a data processing scheme. From the perspective of expert load balancing, the data processing scheme proposes an expert load balancing loss. This expert load balancing loss can be used to characterize the load balancing degree of each expert network in MoE. In other words, the expert load balancing loss can be directly related to the load balancing degree of each expert network in MoE. By training MoE with this expert load balancing loss, the problem of expert load imbalance in MoE can be effectively improved.

[0093] The data processing scheme proposed in this application can be applied to any model that incorporates MoE. For example, the data processing scheme proposed in this application can be applied to models in the field of natural language processing in artificial intelligence technology that incorporate MoE, such as pre-trained language processing models in the field of natural language processing; the data processing scheme proposed in this application can also be applied to models in the field of image processing in artificial intelligence technology that incorporate MoE, such as the ViT (Vision Transformer) model (an image processing model) used for image processing; the data processing scheme proposed in this application can also be applied to models in the field of speech processing in artificial intelligence technology that incorporate MoE, such as the wav2vec model (an audio processing model) used for speech processing; it is worth noting that this application provides a detailed description using the introduction of MoE in a pre-trained language processing model as an example, and the introduction of MoE in models in the fields of image processing and speech processing is similar to that in a pre-trained language processing model.

[0094] Artificial Intelligence (AI) technology refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation.

[0095] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0096] Image processing falls under the field of computer vision. Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs further image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition, as well as liveness detection technology.

[0097] Key technologies in speech technology include automatic speech recognition, speech synthesis, and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech being one of the most promising methods.

[0098] It should be noted that the data processing scheme provided in this application embodiment can be executed by a computer device, which can be a terminal or a server. The server mentioned in this application embodiment can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This application embodiment does not limit this. The terminal mentioned in this application embodiment can be a smartphone, tablet computer, laptop computer, desktop computer, smart voice interaction device, smartwatch, vehicle terminal, smart home appliance, aircraft, etc., but is not limited thereto.

[0099] The data processing scheme provided in the embodiments of this application will be described in more detail below with reference to the accompanying drawings.

[0100] This application provides a data processing method, which mainly describes the construction process of expert load balancing loss. This data processing method can be executed by a computer device, which can be a terminal or a server. Figure 3 As shown, the data processing method may include, but is not limited to, the following steps S301-S304:

[0101] S301 calls the business processing model to perform business processing on the sample data.

[0102] The business processing model introduces an expert sub-model (MoE), meaning that the business processing model can include expert sub-models. The number of expert sub-models included in the business processing model can be one or more, and an expert sub-model can include multiple expert networks. Multiple expert networks belonging to the same expert sub-model can have the same network structure, and the specific processing procedures of each expert network are the same.

[0103] Sample data is used to train the business processing model, and the sample data can include multiple sample units. As mentioned above, the business processing model can be a model from the fields of natural language processing, image processing, or speech processing. Accordingly, for a business processing model in the field of natural language processing, the sample data can be text sample data, and the sample units in the sample data can be words in the text sample data; for a business processing model in the field of image processing, the sample data can be image sample data or video sample data, and the sample units in the sample data can be image patches in the image sample data or video sample data; for a business processing model in the field of audio processing, the sample data can be speech sample data, and the sample units in the sample data can be speech segments in the speech sample data. More specifically, sample data can refer to a batch of sample data, which can include one or more training samples, and each training sample can include multiple sample units; taking text as an example, a training sample can be a training text sentence, which can include multiple words, and a batch of text sample data can include one or more training text sentences, that is, a batch of text sample data can include words from one or more training text sentences.

[0104] S302, during the business processing, obtain the expert allocation information of each sample unit in the sample data in the expert sub-model.

[0105] During the process of calling the business processing model to process the sample data, the expert allocation information of each sample unit in the sample data in the expert sub-model can be obtained. The expert allocation information of any sample unit in the expert sub-model includes: the probability information of the sample unit being assigned to each expert network in the expert sub-model for processing.

[0106] The expert sub-model may also include a gating network. The gating network can be used to perform expert network allocation processing on each sample unit in the sample data to determine the expert allocation information of each sample unit in the expert sub-model. Specifically, for any sample unit in the sample data (which can be represented as a target sample unit), its vector representation can be obtained. Then, the gating network can be invoked to perform expert network allocation processing on the target sample unit based on its vector representation, thus obtaining the expert allocation information of the target sample unit in the expert sub-model. The following sections describe the process of obtaining the vector representation of the target sample unit and the expert network allocation process using the gating network:

[0107] (1) The process of obtaining the vector representation of the target sample unit:

[0108] The vector representation of the target sample unit can be obtained by performing attention aggregation processing on the target sample unit. Specifically, this may include the following steps ①-③:

[0109] ① Obtain the intermediate vector matrix of the target training samples to which the target sample unit belongs.

[0110] As can be seen from the foregoing, the sample data may include one or more training samples, and each training sample may include multiple sample units. The target sample unit is any sample unit in the sample data. The intermediate vector matrix corresponding to the target training sample to which the target sample unit belongs can be obtained. Specifically, the intermediate vector matrix may be composed of the intermediate vectors of each sample unit included in the target training sample.

[0111] In this context, the intermediate vector of a sample unit can refer to the output vector of the sample unit at a certain layer or sub-model of the business processing model during the business processing process. For example, the intermediate vector of a sample unit can be the embedding vector obtained by embedding the sample unit into the input layer of the business processing model. Embedding representation specifically refers to: vector encoding the sample unit to obtain its token embedding vector; adding the token embedding vector to the position embedding vector of the sample unit yields its embedding vector. The position embedding vector of any sample unit can be used to represent its position information in the sample data. For ease of understanding, the length of the training sample, i.e., the number of sample units contained in the training sample, can be represented as L, and the length of the embedding vector of each sample unit can be represented as d (d can take the value 768). Therefore, the size of the intermediate vector matrix X composed of the embedding vectors of each sample unit in the target training sample is L×d, which can be represented as X∈R. L×d .

[0112] ② Perform attention aggregation processing on the intermediate vector matrix to obtain the attention vector matrix.

[0113] Attention aggregation processing can be understood as multi-head attention processing. Specifically, it involves performing multiple attention operations on an intermediate vector matrix and then aggregating the results of these operations (i.e., the attention head information obtained from the multiple attention operations). The purpose of these multiple attention operations is to learn the characteristics represented by the intermediate vector matrix from different perspectives. More specifically, the process of performing attention aggregation processing on the intermediate vector matrix to obtain the attention vector matrix can include:

[0114] The first step involves performing h attention operations on the intermediate vector matrix based on the attention mechanism, obtaining the attention head information corresponding to each of the h attention operations, where h is a positive integer. The i-th attention operation is any one of the h attention operations, where i is a positive integer less than or equal to h. Here, we will use the i-th attention operation out of the h attention operations as an example for explanation:

[0115] First, the intermediate vector matrix can be linearly transformed according to the linear transformation matrix corresponding to the i-th attention operation to obtain a linear vector matrix. There can be one or more linear transformation matrices; one linear transformation matrix corresponds to one linear transformation operation, and one linear transformation operation yields one linear vector matrix. For example, if there are three linear transformation matrices, then three linear transformation operations are required to obtain three linear vector matrices. The process of linear transformation is shown in Equations 4-6 below:

[0116] Q i =XW i q Formula 4

[0117] K i =XW i k Formula 5

[0118] V i =XW i v Formula 6

[0119] In formulas 4-6 above: X represents the intermediate vector matrix; Q i W represents the linear transformation matrix i q The corresponding linear vector matrix, W represents the linear transformation matrix i q It is of size d×d k The matrix, the linear transformation matrix W i q These are learnable model parameters, where d k =d / h; K i W represents the linear transformation matrix i k The corresponding linear vector matrix, W represents the linear transformation matrix i k It is a matrix of size d×dk, and the linear transformation matrix W i k These are learnable model parameters; V i W represents the linear transformation matrix i vThe corresponding linear vector matrix, W represents the linear transformation matrix i v It is of size d×d k The matrix, the linear transformation matrix W i v These are learnable model parameters.

[0120] Then, the linear vector matrix can be normalized to obtain the attention head information corresponding to the i-th attention operation. The normalization process can transform the values ​​into the interval (0, 1). This normalization process can be implemented using the softmax function. The specific process of normalization can be found in the following formula 7:

[0121]

[0122] In formula 7 above: head i This represents the attention head information corresponding to the i-th attention operation.

[0123] The second step involves concatenating the attention head information corresponding to each of the h attention operations to obtain an attention vector matrix. This matrix can include the attention vectors of the sample units in the target training samples. The concatenation process can be implemented using the `concat` function, and the specific steps are detailed in Formula 8 below.

[0124] Multihead=concat(head1, head2,..., head h W o Formula 8

[0125] In Equation 8 above: Multihead represents the attention vector matrix; W o These are learnable model parameters, W o ∈R d×d W o It is a matrix of size d×d.

[0126] ③ The attention vector of the target sample unit in the attention vector matrix is ​​determined as the vector representation of the target sample unit.

[0127] (2) Invoking the expert network allocation process of the gated network:

[0128] The expert network assignment process for a gated network can be implemented using the softmax function, as detailed in Formula 9 below:

[0129] P(x) = softmax(xW) r ) Formula 9

[0130] In Formula 9 above: x represents the vector representation of the target sample unit, x∈R d The vector representation of the target sample unit is a one-dimensional vector of length d; P(x) represents the expert allocation information of the target sample unit under the expert sub-model, where P(x) = (p1(x), p2(x), ..., p... N (x)), p i (x) represents the probability information that the target sample unit is assigned to the i-th expert network in the expert sub-model for processing; W r These are the network parameters of the gated network, which are learnable model parameters, W. r ∈R d×N W r It is a matrix of size d×N.

[0131] S303, based on the expert allocation information of each sample unit in the expert sub-model, determine the expert load balancing loss of the business processing model.

[0132] After obtaining the expert allocation information of each sample unit in the expert sub-model, the expert load balancing loss of the business processing model can be determined based on this information. The expert load balancing loss of the business processing model can be used to characterize the load balancing degree of each expert network in the expert sub-model when each sample unit in the sample data is allocated to the expert network in the expert sub-model for processing according to the corresponding probability information. In other words, the expert load balancing loss of the business processing model is directly related to the load balancing degree of each expert network in the expert sub-model. The more balanced the load of each expert network in the expert sub-model, the smaller the value of the expert load balancing loss of the business processing model, and vice versa.

[0133] The expert load balancing loss of the business processing model can mainly be constructed in two ways: the first is based on information entropy, and the second is based on the importance of the expert network. These two methods will be introduced below:

[0134] (1) Expert load balancing loss construction method based on information entropy:

[0135] ① Calculate the amount of information about expert allocation in the expert sub-model for each sample unit to obtain the amount of information about expert allocation in the expert sub-model for each sample unit.

[0136] Information content calculation can be achieved through information entropy calculation. Taking a target sample unit as an example, the information entropy can be calculated on the probability information of the target sample unit being assigned to each expert network in the expert sub-model for processing, thus obtaining the expert allocation information content of the target sample unit in the expert sub-model. The information entropy calculation process can be seen in the following formula 10:

[0137]

[0138] In Formula 10 above: N represents the number of expert networks included in the expert sub-model; x represents the target sample unit; H(x) represents the amount of expert allocation information of the target sample unit in the expert sub-model; p i (x) represents the probability information that the target sample unit is assigned to the i-th expert network in the expert sub-model for processing.

[0139] ② Determine the expert load balancing loss of the business processing model based on the difference between the amount of expert-assigned information in each sample unit in the expert sub-model and the amount of information equally assigned in the expert sub-model.

[0140] The information content of equal allocation in an expert sub-model refers to the amount of information when sample units are allocated to each expert network in the expert sub-model with equal probability for processing. The calculation process for equal allocation information can be found in Formula 11 below:

[0141]

[0142] In formula 11 above: H μ (x) represents the equal distribution of information in the expert sub-model; N represents that the expert sub-model includes N expert networks; This indicates that the probability information of a sample unit being assigned to each expert network within an expert subnetwork for processing is equal, and is 1 / 2.

[0143] When the number of expert sub-models included in the business processing model is one, the process of determining the expert load balancing loss of the business processing model may include: calculating the information content difference of each sample unit in the expert sub-model based on the difference between the expert-allocated information content of each sample unit in the expert sub-model and the equally allocated information content of the expert sub-model; then, the average of the information content differences of each sample unit in the expert sub-model can be calculated to obtain the expert load balancing loss of the business processing model. See Formula 12 below for details:

[0144]

[0145] As shown in formula 12 above: L auxB represents the expert load balancing loss of the business processing model; T represents the sample data of the current batch; and T represents the number of sample units contained in the sample data of the current batch.

[0146] When the number of expert sub-models included in the business processing model is multiple, the process of determining the expert load balancing loss of the business processing model may include: for any expert sub-model (i.e., the target expert sub-model) included in the business processing model, the information content difference of each sample unit in the target expert sub-model can be calculated based on the difference between the expert allocation information content of each sample unit in the target expert sub-model and the equal allocation information content of the target expert sub-model; then, the average of the information content differences of each sample unit in the target expert sub-model can be calculated to obtain the expert load balancing loss of the target expert sub-model; and the expert load balancing losses of each expert sub-model included in the business processing model can be summed to obtain the expert load balancing loss of the business processing model.

[0147] In summary, the expert load balancing loss construction method based on information entropy has the following advantages: the number of expert networks included in the expert sub-model has no impact on the construction process of the expert load balancing loss; therefore, this embodiment can support including any number of expert networks in the expert sub-model. The number of expert sub-models included in the business processing model has no impact on the construction process of the expert load balancing loss; therefore, this embodiment can support introducing any number of expert sub-models into the business processing model. The expert load balancing loss is directly related to the load balancing degree of each expert network in the expert sub-model. Therefore, training the business processing model based on expert load balancing loss can effectively improve the load imbalance problem among expert networks in the expert sub-models included in the business processing model. The expert load balancing loss is constructed by the difference between the information allocated by experts and the information allocated equally. When the expert networks in the expert sub-models achieve load balancing, the information allocated by experts is equal to the information allocated equally. In other words, when the expert networks in the expert sub-models achieve load balancing, the expert load balancing loss is 0. This will not hinder the optimization of the model for the main objective, which is the business processing loss generated by calling the business processing model to process the sample data.

[0148] (2) Expert load balancing loss construction method based on expert network importance:

[0149] ① Based on the expert allocation information of each sample unit in the expert sub-model, calculate the importance of each expert network in the expert sub-model. The importance of any expert network can be defined as the cumulative sum of the probability information of each sample unit in the sample data being assigned to that expert network for processing. See Formula 1 above for details.

[0150] ② Calculate the scaling factor of each expert network in the expert sub-model. The scaling factor of any expert network can be defined as: the number of sample units in the sample data allocated to that expert network for processing divided by the total number of sample units in the sample data. The scaling factor f of the i-th expert network... i This can be described as Formula 13:

[0151]

[0152] In the above formula 13: P(x) = (p1(x), p2(x), ..., p N (x) represents the probability information that the gating network assigns sample unit x in the sample data to N expert networks for processing, that is, the expert assignment information of sample unit x in the expert sub-network; δ represents the Kronecker function, which can be defined as follows: Formula 14:

[0153]

[0154] ③ The proportional factors of each expert network in the expert sub-model can be used as weights to perform a weighted summation of the importance of each expert network, thus obtaining the expert load balancing loss of the business processing model. See Formula 15 below for details:

[0155]

[0156] In Formula 15 above: α is a hyperparameter that can be manually adjusted to control the weight of the loss, and is usually set to α = 0.01; Importance i This represents the importance of the i-th expert network.

[0157] The above ①-③ describe the case where the business processing model includes one expert sub-model. When the business processing model includes multiple expert sub-models, for any expert sub-model (i.e., the target expert sub-model), the importance of each expert network in the target expert sub-model can be calculated based on the expert allocation information of each sample unit in the target expert sub-model. The scaling factor of each expert network in the target expert sub-model can then be calculated. Then, the scaling factor of each expert network in the target expert sub-model can be used as a weight to perform a weighted summation of the importance of each expert network in the target expert sub-model to obtain the expert load balancing loss of the target expert sub-model. Finally, the expert load balancing losses of each expert sub-model included in the business processing model can be summed to obtain the expert load balancing loss of the business processing model.

[0158] In summary, the expert load balancing loss construction method based on expert network importance described above has the following advantages: the number of expert networks included in the expert sub-model has no impact on the construction process of the expert load balancing loss; therefore, this embodiment can support the inclusion of any number of expert networks in the expert sub-model. The number of expert sub-models introduced into the business processing model has no impact on the construction process of the expert load balancing loss; therefore, this embodiment can support the introduction of any number of expert sub-models into the business processing model. The expert load balancing loss is directly related to the load balancing degree of each expert network in the expert sub-model; therefore, training the business processing model based on the expert load balancing loss can effectively improve the load imbalance problem among the expert networks in the expert sub-models included in the business processing model. However, the expert load balancing loss constructed based on the importance of expert networks has the drawback of non-zero property. Non-zero property means that when all expert networks in the expert sub-model reach load balancing, the expert load balancing loss is equal to 1 / N and not equal to 0. This non-zero property will hinder the optimization of the model for the main objective. Because when all expert networks in the expert sub-model reach load balancing, the business processing model should only focus on the optimization of the main objective. However, the non-zero property makes the business processing model still optimize the expert load balancing loss. At this time, there is no possibility of further optimization of expert load balancing. Further optimization will hinder the optimization of the main objective.

[0159] S304. Train the business processing model according to the rule of reducing the expert load balancing loss of the business processing model.

[0160] After determining the expert load balancing loss of the business processing model, the business processing model can be trained according to the rules for reducing the expert load balancing loss of the business processing model, so that the various expert networks in the expert load network included in the business processing model can achieve load balancing.

[0161] In this embodiment, the business processing model can incorporate any number of expert sub-models, and each expert sub-model can include any number of expert networks. Furthermore, the constructed expert load balancing loss is directly related to the load balancing degree of each expert network in the expert sub-model. By training the business processing model using the constructed expert load balancing loss, the load imbalance problem among the expert networks in the expert sub-models included in the business processing model can be effectively improved. In addition, when each expert network in the expert sub-models included in the business processing model achieves load balancing, the expert load balancing loss constructed based on information entropy is equal to zero. Thus, when the expert sub-models achieve load balancing, it will not affect the optimization process of the business processing model with respect to the main objective.

[0162] This application provides a data processing method, which mainly describes the process of constructing a business processing loss (business processing loss refers to the loss generated when calling a business processing model to perform business processing on sample data). This data processing method can be executed by a computer device, which can be a terminal or a server. Figure 4 As shown, the data processing method may include, but is not limited to, the following steps S401-S406:

[0163] S401 calls the business processing model to perform business processing on the sample data.

[0164] In this embodiment of the application, the execution process of step S401 is the same as described above. Figure 3 The execution process of step S301 in the illustrated embodiment is the same, and can be found in the above description. Figure 3 The relevant description of step S301 in the illustrated embodiment will not be repeated here.

[0165] S402, during the business processing, obtain the expert allocation information of each sample unit in the sample data in the expert sub-model.

[0166] In this embodiment, the execution process of step S402 is the same as described above. Figure 3 The execution process of step S302 in the illustrated embodiment is the same; please refer to the above for details. Figure 3 The relevant description of step S302 in the illustrated embodiment will not be repeated here.

[0167] S403, based on the expert allocation information of each sample unit in the expert sub-model, determine the expert load balancing loss of the business processing model.

[0168] In this embodiment of the application, the execution process of step S403 is the same as described above. Figure 3 The execution process of step S303 in the illustrated embodiment is the same; please refer to the above for details. Figure 3The relevant description of step S303 in the illustrated embodiment will not be elaborated here.

[0169] S404. During the business processing, obtain the business processing loss of the business processing model for the sample data.

[0170] The business processing process of the business processing model for the sample data can specifically refer to: the content understanding process of the business processing model for the sample data, that is, the process of identifying and understanding the content expressed by the sample data. In the embodiment of the present application, taking the business processing model as a pre-trained language model in the field of natural language processing and the sample data as text sample data, the process of the business processing model performing business processing on the sample data is introduced.

[0171] Before introducing the process of the business processing model performing business processing on the sample data, the construction process of the sample data will be introduced here. Specifically, the following content can be referred to: The original text samples for training the business processing model can be obtained. The original text samples can include multiple sample units; then, some of the sample units in the original text samples can be replaced by replacement units (also called "sentinel tokens"), and the input text samples are obtained. The replacement units are used to splice the replaced sample units to obtain the output text samples. The replaced sample units can be randomly selected from the original text samples according to a certain probability (for example, 15%); then, the input text samples and the output text samples can be combined to obtain the training samples, that is, the training samples are text sample pairs (input text samples, output text samples) composed of the input text samples and the output text samples. The sample data (that is, a batch of sample data) can include one or more training samples.

[0172] As Figure 5 shown, the original sample text is "Science and technology are the primary productive forces". The replaced sample units in the original sample text are the four characters "technology", "technique", "production", and "productivity". The continuously arranged sample units among the replaced sample units can be regarded as one sample unit, and the non-continuously arranged sample units can be regarded as separate sample units. Thus, the continuously arranged sample unit "technology" uses the replacement unit <x>Replacement is performed; consecutively arranged sample units are "produced" using replacement units. <y>The replacement is performed to obtain the input text sample; replacement unit <x>The replaced sample unit "technique" is spliced ​​together, and the replacement unit is replaced. <y>By splicing the replaced sample unit "production", the output text sample is obtained, thus completing the construction of a training sample.

[0173] Before introducing the business processing model's business processing procedures for sample data, let's first combine... Figure 6 The structure of the business processing model is described below. The business processing model may include:

[0174] (1) Input layer: The input layer can be used to embed each input sample unit in the input text sample to obtain the embedding vector of each input sample unit in the input text sample.

[0175] (2) Encoding Model (Encoder): The encoding model can be used to encode each input sample unit in the input text sample to obtain the encoding result of each input sample unit. The encoding model can include one or more encoding blocks (EncoderTransformerBlock), and each encoding block has the same structure and the same encoding process. When the encoding model includes multiple encoding blocks, the multiple encoding blocks are sequentially encoded. Sequential encoding process can specifically mean that the second encoding block further encodes the encoding result of the first encoding block to obtain the encoding result of the second encoding block, the third encoding block further encodes the encoding result of the second encoding block to obtain the encoding result of the third encoding block, and so on, until the encoding process obtains the encoding result of the last encoding block.

[0176] Any coded block may include the following three parts:

[0177] ① Multi-head Attention: Used for attention aggregation processing.

[0178] ② Expert Sub-model (MoE Layer): The expert sub-model can be formed by extending the FNN as an expert network into a multi-expert network.

[0179] ③ Residual Standardization Layers (Add & Norm): The encoding block can include two residual standardization layers. The first residual standardization layer is located between the multi-head attention layer and the expert sub-model, used to perform residual connection (specifically, addition) on the input and output of the multi-head attention layer before standardization. The second residual standardization layer is located between the expert sub-model and the next encoding block, used to perform residual connection (specifically, addition) on the input and output of the expert sub-model before standardization. Standardization specifically refers to adjusting the input to a suitable range based on its mean and variance. Residual connection effectively reduces model complexity during training to minimize overfitting, while standardization avoids the vanishing gradient problem during training, accelerating model convergence.

[0180] (3) Decoder Model: The decoder model can be used to decode the encoding results of the encoding model. Specifically, the decoding process can be a word prediction process. During the training of the business processing model, the decoder model can be used to combine the encoding results of each input sample unit in the input text sample with the encoding model to perform word prediction on each output sample unit in the output text sample. During the application of the business processing model, the decoder model can be used to combine the encoding results of each text in the text to be processed with the encoding model to perform content tag prediction on the text to be processed. Similar to the encoding model, the decoder model can include one or more decoder blocks (DecoderTransformerBlock). Each decoder block has the same structure and the same decoding process. When the decoder model includes multiple decoder blocks, multiple decoder blocks undergo successive decoding processing. Specifically, successive decoding processing can refer to: the second decoder block further decoding the decoding result of the first decoder block to obtain the encoding result of the second decoder block; the third decoder block further decoding the decoding result of the second decoder block to obtain the decoding result of the third decoder block; and so on, until the decoding process yields the decoding result of the last decoder block. Any decoder block can include the following four parts:

[0181] ① Multi-head Attention: Used for attention aggregation processing.

[0182] ② Masked Multi-head Attention: The logic of the masked multi-head attention layer is similar to that of the multi-head attention layer. The purpose of masking is that when predicting a word, one sample unit is predicted at a time. When the current sample unit is predicted, the sample units after the current sample unit need to be masked, otherwise it will affect the current sample unit.

[0183] ③ Expert Sub-model (MoE Layer): The expert sub-model can be formed by extending the FNN as an expert network into a multi-expert network.

[0184] ④ Residual Normalization Layers (Add & Norm): A decoding block may include three residual normalization layers. The first residual normalization layer in the decoding block is located between the masked multi-head attention layers and is used to perform residual connection and normalization on the input and output of the masked multi-head attention layers. The second residual normalization layer in the decoding block is located between the multi-head attention layers and the expert sub-model and is used to perform residual connection and normalization on the input and output of the multi-head attention layers. The third residual normalization layer in the decoding block is located between the expert sub-model and the next decoding block (which is the output layer for the last decoding block) and is used to perform residual connection and normalization on the input and output of the expert sub-model.

[0185] (4) Output layer: During the model training phase, the output layer can be used to calculate the business processing loss based on the difference between the predicted results of the decoding model and the actual results; during the model application phase, the output layer can be used to output the predicted content labels.

[0186] Based on the structure of the above business processing model, the process of calling the business processing model to perform business processing on the sample data can specifically include the following sub-steps s71-s74:

[0187] s71, training initialization.

[0188] Training initialization can include initializing the model parameters, optimizer, learning rate, and training steps (Train_step) of the business processing model. Specifically, model parameter initialization can refer to: using random numbers sampled from a truncated Gaussian distribution N(μ, σ) as the initial model parameters, where μ = 0 represents the mean of the Gaussian distribution and σ = 0.01 represents the standard deviation of the Gaussian distribution; initializing the optimizer to an Adam optimizer with parameters β1 = 0.9 and β2 = 0.998; and setting the learning rate to 1 × e^(-1 / 2). -4 The training steps are initialized to Train_step = 1.

[0189] s72 calls the input layer to embed the training samples.

[0190] Specifically, the training samples can include input text samples and output text samples. The input layer can be called to embed each input sample unit in the input text sample to obtain the embedding vector of each input sample unit. Similarly, the input layer can be called to embed each output sample unit in the output text sample to obtain the embedding vector of each output sample unit.

[0191] like Figure 7 As shown, the input text sample is "science". <x>It is the first <y>Taking "force" as an example, embedding representation can specifically refer to: performing vector encoding on the input sample unit to obtain the token embedding vector of the input sample unit, adding the token embedding vector of the input sample unit to the position embedding vector of the input sample unit, and then obtaining the embedding vector of the input sample unit. The position embedding vector of any input sample unit can be used to represent the position information of the sample unit in the sample data.

[0192] s73, call the encoding model to encode each input sample unit in the input text sample, and obtain the encoding result of each input sample unit in the input text sample.

[0193] Specifically, the embedding vectors of each input sample unit in the input text sample can be input into the encoding model, and the encoding model can be called to encode the embedding vectors of each input sample unit in the input text sample to obtain the encoding results of each input sample unit in the input text sample.

[0194] Taking an encoding model consisting of a single encoding block as an example, the encoding process of the encoding model may include: calling a multi-head attention layer to perform attention aggregation processing on the embedding vectors of each input sample unit in the input text sample, obtaining the attention vector of each input sample unit in the input text sample; calling a first residual normalization layer to perform residual normalization processing on the input and output of the multi-head attention layer, obtaining the output vector of each input sample unit in the input text sample with respect to the first residual normalization layer (i.e., the vector representation of each input sample unit in the input text sample); calling an expert sub-model to process the vector representation of each input sample unit in the input text sample, obtaining the processing result of the expert sub-model on each input sample unit in the input text sample; and calling a second residual normalization layer to perform residual annotation processing on the processing result of the expert sub-model, obtaining the encoding result of each input sample unit in the input text sample.

[0195] Taking any input sample unit (which can be represented as a reference input sample unit) in the input text sample as an example, the processing of the reference input sample unit by the expert sub-model can include:

[0196] First, based on the expert allocation information of the reference input sample unit in the expert sub-model, one or more target expert networks can be determined in the expert sub-model to perform calculations on the reference input sample unit. Specifically, the expert allocation information of the reference input sample unit in the expert sub-model can include: the probability information of the reference input sample unit being assigned to various expert networks in the expert sub-model for processing. Following a Top-k routing strategy, the k expert networks with the highest probability information can be selected from the expert sub-model, and these k selected expert networks can be used as the target expert networks. Figure 8 As shown, the current batch of sample data includes 3 training samples, that is, the current batch of sample data includes 3 input text samples, a i b i c i (i = 1, 2, 3, 4, 5) represent input sample units belonging to different input text samples. Input sample units of the same color belong to the same input text sample. For example, (a1, a2, a3, a4, a5) represents one input text sample. The expert sub-model includes 5 expert networks. For input sample unit a1, among the probability information of input sample unit a1 being assigned to the 5 expert networks for processing, the probability information P2 of input sample unit a1 being assigned to the 2nd expert network for processing is the largest, followed by the probability information P5 of input sample unit a1 being assigned to the 5th expert network for processing. If the expert network with the highest probability information is selected from the expert sub-model, then the selected expert network for processing input sample unit a1 is the 2nd expert network.

[0197] Secondly, a selected target expert network can be used to calculate the reference input sample unit, and the calculation result of the target expert network for the reference input sample unit can be obtained; the calculation process of any target expert network (which can be represented as the i-th target expert network) for the reference input sample unit x can be seen in the following formula 16:

[0198]

[0199] In Equation 16 above: x represents the reference input sample cell; E i (x) represents the computation result of the i-th target expert network on the reference input sample unit; FFN represents the feedforward neural network; W i 1 W is a trainable model parameter in an FFN network. i 1 ∈R d×4d W i 1 W is a matrix of size d×4d; i 2 W is a trainable model parameter in an FFN network. i 2 ∈R 4d×d W i 2 It is a matrix of size 4d×d; It is a trainable model parameter in the FFN network. express It is a one-dimensional vector of length 4d. It is a trainable model parameter in the FFN network. express It is a one-dimensional vector of length d.

[0200] Then, based on the probability information of the reference input sample units being assigned to the target expert network for computation, the computation results of the target expert network on the reference input sample units can be weighted and summed to obtain the processing result of the expert sub-model on the reference input sample units. See Formula 17 below for details:

[0201] y = ∑ i∈τ p i (x)E i (x) Formula 17

[0202] In Formula 17 above: τ represents the set of selected target expert networks; p i (x) represents the probability information of a reference input sample unit being assigned to the i-th target expert network for processing; E i (x) represents the processing result of the i-th target expert network on the reference input sample unit; y represents the processing result of the expert sub-model on the reference input sample unit.

[0203] s74 calls the decoding model to combine the encoding results of each input sample unit in the input text sample to perform word prediction on each output sample unit in the output text sample, and obtains the probability distribution information of each output sample unit in the output text sample in the labeled vocabulary.

[0204] Specifically, a decoding model can be invoked to combine the encoding results of each input sample unit in the input text sample to decode each output sample unit in the output text sample, obtaining the decoding results of each output sample unit in the output text sample. The decoding process of the decoding model is similar to the encoding process of the encoding model, and will not be elaborated here. Then, based on the decoding results of each output sample unit in the output text sample, word prediction can be performed on each output sample unit in the output text sample to obtain the probability distribution information of each output sample unit in the labeled vocabulary. The probability distribution information of any output sample unit in the output text sample in the labeled vocabulary can include: the probability that the output sample unit is predicted to be any word in the labeled vocabulary.

[0205] Specifically, the vocabulary prediction processing can refer to: performing a linear transformation on the decoding results of each output sample unit in the output text sample, followed by normalization; the linear transformation can be implemented using a Linear function (a type of linear transformation function), and the normalization can be implemented using a softmax function, as shown in Formula 18 below:

[0206]

[0207] In Formula 18 above: x represents any output sample unit in the output text sample (which can be represented as a reference output sample unit); This represents the probability distribution information of the reference output sample unit in the labeled vocabulary. W represents the probability that the reference output sample unit is predicted to be the i-th word in the labeled vocabulary; v W is a trainable model parameter. v ∈R d×|v| W v It is a matrix of size d×|v|; |v| represents the size of the labeled vocabulary, that is, the number of words contained in the labeled vocabulary.

[0208] After obtaining the probability distribution information of each output sample unit in the labeled vocabulary through vocabulary prediction, the business processing loss can be calculated based on the probability distribution information of each output sample unit in the labeled vocabulary and the labeled position vector of each output sample unit in the labeled vocabulary. Specifically, the labeled position vector of any output sample unit (which can be represented as a reference output sample unit) in the labeled vocabulary can be used to indicate the position information of the real word represented by the reference output sample unit in the labeled vocabulary. The labeled position vector of the reference output sample unit in the labeled vocabulary can be represented as P. * (x)∈R |v| The P * (x) is a one-hot vector of length |v|, where the values ​​in the vector that indicate the position of the real word represented by the reference output sample unit in the labeled vocabulary are 1, and the rest are 0; the calculation process of the processing loss for any output sample unit (which can be represented as the reference output sample unit) can be found in the following formula 19:

[0209]

[0210] In Equation 19 above: Loss(x) represents the processing loss of the reference output sample unit x; P * (x) represents the annotation position vector of the reference output sample unit x in the annotation vocabulary; This represents the probability distribution information of the reference output sample unit in the labeled vocabulary. After calculating the processing loss of each output sample unit contained in the output text sample in the sample data, the processing loss of each output sample unit can be summed to obtain the business processing loss, as shown in Formula 20 below:

[0211]

[0212] In formula 20 above: L output The value represents the processing loss; Loss(x) represents the processing loss of any output sample unit x; B represents the sample data of the current batch; T represents the number of output sample units contained in the sample data of the current batch.

[0213] S405 sums the business processing loss and the expert load balancing loss of the business processing model to obtain the loss information of the business processing model.

[0214] After obtaining the business processing loss and the expert load balancing loss of the business processing model, the business processing loss and the expert load balancing loss of the business processing model can be summed to obtain the loss information of the business processing model. The specific calculation process of the loss information of the business processing model can be found in the following formula 21:

[0215] L model =L output +λL aux Formula 21

[0216] In formula 21 above: L model This represents the loss information of the business processing model; L output Indicates business processing loss; L aux λ represents the expert load balancing loss of the business processing model; λ represents a hyperparameter used to control the expert load balancing loss of the business processing model, which is usually 0.1, 0.01 or 0.001.

[0217] S406. Train the business processing model according to the rule of reducing the loss information of the business processing model.

[0218] After obtaining the loss information of the business processing model, the model can be trained according to rules that reduce this loss information. Specifically, these rules refer to optimizing the model in a direction that minimizes the loss information. The optimized model should produce a loss less than the original loss information. For example, if the calculated loss information of the business processing model is 0.85, then optimizing the model in this way should result in a loss less than 0.85.

[0219] After training the business processing model using the current batch of sample data, it can be determined whether the training termination condition has been met. If the training termination condition is met, training of the business processing model is stopped, and a trained business processing model is obtained. If the training termination condition is not met, the next batch of sample data can be obtained, and training of the business processing model can continue. The determination of whether the training termination condition has been met can be done in two ways: The first way is to check whether the current training steps have reached the set maximum number of steps (MAX_STEP, which is usually set to 1,000,000). If the maximum number of steps has been reached, the training termination condition is determined to have been met; if not, the training termination condition is determined not to have been met, and the training steps are updated (e.g., Training_step = Training_step + 1). The second way is to check whether the current loss information is less than the loss threshold. If it is less than the loss threshold, the training termination condition is determined to have been met; if it is greater than or equal to the loss threshold, the training termination condition is determined not to have been met.

[0220] The trained business processing model can be used to perform content recognition or content understanding on media data, obtaining content tags for the media data. Media data can include, but is not limited to, any of the following: text data, image data, video data, and audio data, etc. The process of calling the trained business processing model to perform content recognition on media data is similar to the process of calling the business processing model to perform business processing on sample data described above, and will not be repeated here. The identified content tags can be used in scenarios such as media data classification and related content recommendation. Figure 9 This demonstrates a scenario where content tags are applied to ad recommendations. The trained business processing model receives a text input: "A reasonable practice process and time allocation table for piano children aged 3-12, worth learning from!" The model recognizes and understands that the text is related to children's music, and therefore outputs the content tag "Quality Education | Children's Music". This content tag is applied to the object feature module, coarse ranking feature module, and fine ranking feature module in the advertising system, thereby improving the accuracy of the coarse ranking model and the fine ranking model, and ultimately improving the accuracy of the ad system's recommendations.

[0221] In this embodiment of the application, for the business processing loss of the introduced expert sub-model, the business processing loss of the business processing model and the expert load balancing loss of the business processing model can be combined to train the business processing model. On the one hand, it can improve the business processing capability of the business processing model, and on the other hand, it can effectively improve the load imbalance problem of each expert network in the expert sub-model included in the business processing model, thus greatly improving the training effect of the business processing model.

[0222] The following describes some experimental results of the embodiments of this application in specific experimental scenarios:

[0223] (1) Based on the loss convergence level index, compare the training effect of the business processing model using the expert load balancing loss based on information entropy (hereinafter referred to as the first loss construction method) with the training effect of the expert load balancing loss based on expert network importance (hereinafter referred to as the second loss construction method). Figure 10 The diagram shows the loss curves of the training set when training the business processing model using two different loss construction methods. The black curve represents the loss curve of the training set when training the business processing model using the first loss construction method, and the gray curve represents the loss curve of the training set when training the business processing model using the second loss construction method. Figure 11 The diagram shows the loss curves on the validation set for two different loss construction methods when training the business processing model. The black curve represents the loss curve on the validation set using the first loss construction method, and the gray curve represents the loss curve on the validation set using the second loss construction method. The comparison clearly shows that the first loss construction method converges to a lower level on both the training and validation sets compared to the second method. Therefore, the loss constructed using the first method performs better than the loss constructed using the second method for training the business processing model.

[0224] (2) Based on the perplexity index of the test set, the training effect of the business processing model using the expert load balancing loss constructed based on information entropy (hereinafter referred to as the first loss construction method) is compared with that of the expert load balancing loss constructed based on expert network importance (hereinafter referred to as the second loss construction method). Perplexity (PPL) is an indicator in the field of natural language processing to measure the quality of a language model; the lower the perplexity, the better the language model. Table 1 below shows the performance of the first and second loss construction methods on the test set perplexity when training the business processing model:

[0225] Table 1

[0226] Second loss construction method 5.107 First loss construction method 4.946

[0227] As can be seen from the comparison, the PPL on the test set of the first loss construction method is about 3% lower than that of the second loss construction method when training the business processing model, indicating that the model trained by the first loss construction method has better generalization ability.

[0228] (3) The business processing model proposed in this application embodiment is compared with the baseline model based on classification performance metrics. The baseline model can refer to a model that does not introduce an expert subnetwork. In actual use scenarios, the business processing model performs content recognition on media data to obtain content tags for the media data. The content tags can be used to classify the media data. Classification performance metrics can include at least one of the following: precision (P) metric, recall (R) metric, and F1 metric. The higher the value of the classification performance metric, the better the classification performance of the model. Table 2 below shows the classification performance of the business processing model and the baseline model in different classification tasks:

[0229] Table 2

[0230]

[0231]

[0232] As can be seen from the comparison, compared with the baseline model that does not introduce expert sub-models, the business processing model proposed in this application embodiment has improved in terms of accuracy, recall and F1 score. The content tags identified by the business processing model proposed in this application embodiment have a high classification accuracy in classification tasks.

[0233] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0234] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. The data processing device can be installed in the computer equipment provided in the embodiment of this application. The computer equipment can be the server or terminal mentioned in the above method embodiment. Figure 12 The data processing apparatus shown may be a computer program (including program code) running on a computer device, which can be used to execute... Figure 3 or Figure 4 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 12 The data processing apparatus may include the following units:

[0235] The processing unit 1201 is used to call the business processing model to perform business processing on the sample data; the business processing model includes an expert sub-model, the expert sub-model includes multiple expert networks, and the sample data includes multiple sample units.

[0236] The acquisition unit 1202 is used to acquire the expert allocation information of each sample unit in the sample data in the expert sub-model during the business processing; the expert allocation information includes the probability information of the sample unit being assigned to each expert network in the expert sub-model for processing.

[0237] The processing unit 1201 is also used to determine the expert load balancing loss of the business processing model based on the expert allocation information of each sample unit in the expert sub-model. The expert load balancing loss of the business processing model is used to characterize the load balancing degree of each expert network in the expert sub-model when each sample unit in the sample data is allocated to the expert network in the expert sub-model for processing according to the corresponding probability information.

[0238] The processing unit 1201 is also used to train the business processing model according to the rule of reducing the expert load balancing loss of the business processing model; the trained business processing model is used to identify and understand the content expressed by the media data.

[0239] In one implementation, when processing unit 1201 determines the expert load balancing loss of the business processing model based on the expert allocation information of each sample unit in the expert sub-model, it specifically performs the following steps:

[0240] The information content of expert allocation information for each sample unit in the expert sub-model is calculated to obtain the information content of expert allocation for each sample unit in the expert sub-model.

[0241] Based on the difference between the amount of expert-assigned information in each sample unit in the expert sub-model and the amount of information equally assigned in the expert sub-model, the expert load balancing loss of the business processing model is determined.

[0242] The information content of equal distribution in the expert sub-model refers to the amount of information when sample units are distributed with equal probability to each expert network in the expert sub-model for processing.

[0243] In one implementation, the business processing model includes multiple expert sub-models, and the target expert sub-model is any one of the expert sub-models included in the business processing model. The processing unit 1201, when determining the expert load balancing loss of the business processing model based on the difference between the amount of expert-assigned information in each sample unit within the expert sub-model and the amount of information equally assigned to each expert sub-model, specifically performs the following steps:

[0244] Based on the difference between the amount of information allocated by experts to each sample unit in the target expert sub-model and the amount of information allocated equally in the target expert sub-model, calculate the information content difference of each sample unit in the target expert sub-model.

[0245] The average of the information content difference of each sample unit in the target expert sub-model is calculated to obtain the expert load balancing loss of the target expert sub-model.

[0246] The expert load balancing loss of the business processing model is obtained by summing the expert load balancing losses of each expert sub-model included in the business processing model.

[0247] In one implementation, the target sample unit is any sample unit in the sample data; the expert sub-model also includes a gating network; the acquisition unit 1202 is used to perform the following steps when acquiring the expert allocation information of each sample unit in the sample data in the expert sub-model during business processing:

[0248] Obtain the vector representation of the target sample unit;

[0249] By invoking the gating network and performing expert network allocation processing on the target sample unit based on its vector representation, the expert allocation information of the target sample unit in the expert sub-model is obtained.

[0250] In one implementation, the sample data includes one or more training samples, each training sample including multiple sample units, and the target sample unit is any sample unit in the sample data; the acquisition unit 1202, when acquiring the vector representation of the target sample unit, specifically performs the following steps:

[0251] Obtain the intermediate vector matrix corresponding to the target training sample to which the target sample unit belongs; the intermediate vector matrix is ​​composed of the intermediate vectors of the sample units included in the target training sample;

[0252] The intermediate vector matrix is ​​subjected to attention aggregation processing to obtain the attention vector matrix; the attention vector matrix includes the attention vectors of the sample units in the target training samples;

[0253] The attention vector of the target sample unit in the attention vector matrix is ​​determined as the vector representation of the target sample unit.

[0254] In one implementation, the acquisition unit 1202 is used to perform attention aggregation processing on the intermediate vector matrix to obtain the attention vector matrix, specifically by executing the following steps:

[0255] Based on the attention mechanism, h attention operations are performed on the intermediate vector matrix to obtain the attention head information corresponding to each attention operation in the h attention operations, where h is a positive integer;

[0256] The attention head information corresponding to each attention operation in h attention operations is concatenated to obtain the attention vector matrix.

[0257] In one implementation, the i-th attention operation is any one of the h attention operations, where i is a positive integer less than or equal to h; the acquisition unit 1202, when performing h attention operations on the intermediate vector matrix based on the attention mechanism to obtain the attention head information corresponding to each of the h attention operations, specifically performs the following steps:

[0258] The intermediate vector matrix is ​​linearly transformed based on the linear transformation matrix corresponding to the i-th attention operation to obtain a linear vector matrix.

[0259] Normalize the linear vector matrix to obtain the attention head information corresponding to the i-th attention operation.

[0260] In one implementation, the acquisition unit 1202 is also used to acquire the business processing loss of the business processing model on the sample data during the business processing process.

[0261] Processing unit 1201, when training the business processing model according to the rules for reducing the expert load balancing loss of the business processing model, specifically performs the following steps:

[0262] The loss information of the business processing model is obtained by summing the business processing loss and the expert load balancing loss of the business processing model.

[0263] The business processing model is trained according to the rules for reducing the loss information of the business processing model.

[0264] In one implementation, the business processing model includes an encoding model and a decoding model; the sample data includes input text samples and output text samples, wherein the input text samples include multiple input sample units and the output text samples include multiple output sample units; the acquisition unit 1202 is used to acquire the business processing loss of the business processing model on the sample data during the business processing process, specifically by performing the following steps:

[0265] The encoding model is invoked to encode each input sample unit in the input text sample, and the encoding result of each input sample unit in the input text sample is obtained.

[0266] The decoding model is invoked to combine the encoding results of each input sample unit to perform word prediction on each output sample unit in the output text sample, thereby obtaining the probability distribution information of each output sample unit in the labeled vocabulary. The probability distribution information of the output sample unit in the labeled vocabulary includes the probability that the output sample unit is predicted as each word in the labeled vocabulary.

[0267] The business processing loss is calculated based on the probability distribution information of each output sample unit in the labeled vocabulary and the labeled position vector of each output sample unit in the labeled vocabulary.

[0268] In one implementation, the encoding model includes an expert sub-model; the reference input sample unit is any input sample unit in the input text sample; the acquisition unit 1202 is used to call the encoding model to encode each input sample unit in the input text sample, and when obtaining the encoding result of each input sample unit in the input text sample, it is specifically used to perform the following steps:

[0269] Obtain the processing results of the expert sub-model on the reference input sample unit;

[0270] The residual standardization process is applied to the processing results of the expert sub-model on the reference input sample unit to obtain the encoding result of the reference input sample unit.

[0271] In one implementation, when the acquisition unit 1202 acquires the processing result of the expert sub-model on the reference input sample unit, it specifically performs the following steps:

[0272] Based on the expert assignment information of the reference input sample unit in the expert sub-model, determine one or more target expert networks in the expert sub-model for computing the reference input sample unit;

[0273] The target expert network is used to perform calculations on the reference input sample units, and the calculation results of the target expert network on the reference input sample units are obtained.

[0274] Based on the probability information of the reference input sample units being assigned to the target expert network for computation, the computation results of the target expert network on the reference input sample units are weighted and summed to obtain the processing results of the expert sub-model on the reference input sample units.

[0275] In one implementation, the acquisition unit 1202 is also used to acquire original text samples for training the business processing model, the original text samples including multiple sample units.

[0276] The processing unit 1201 is also used to replace some sample units in the original text sample with a replacement unit to obtain an input text sample;

[0277] The processing unit 1201 is also used to concatenate the replaced sample unit with the replacement unit to obtain the output text sample;

[0278] The processing unit 1201 is also used to combine the input text samples and the output text samples to obtain training samples; the sample data includes one or more training samples.

[0279] In one implementation, the acquisition unit 1202 is further configured to acquire media data to be processed, the media data including any of the following: text, image, video, and audio;

[0280] The processing unit 1201 is also used to call the trained business processing model to perform content recognition on the media data and obtain the content tags corresponding to the media data.

[0281] According to another embodiment of this application, Figure 12 The data processing apparatus shown can be constructed by combining each unit individually or entirely into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0282] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 3 or Figure 4 The computer program (including program code) for each step involved in some or all of the methods shown, to construct such... Figure 12 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computing device via the computer-readable storage medium, and executed therein.

[0283] In this application embodiment, an expert load balancing loss is proposed that can be used to characterize the load balancing degree of each expert network in the expert sub-model (i.e., MoE). In other words, the expert load balancing loss is directly related to the load balancing degree of each expert network in the expert sub-model. Therefore, by training the business processing model containing the expert sub-model with the expert load balancing loss, the load imbalance problem of each expert network in the expert sub-model included in the business processing model can be effectively improved.

[0284] Based on the above methods and apparatus embodiments, this application provides a computer device, which may be the aforementioned server or terminal. Please refer to... Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 13 The computer device shown includes at least a processor 1301, an input interface 1302, an output interface 1303, and a computer-readable storage medium 1304. The processor 1301, input interface 1302, output interface 1303, and computer-readable storage medium 1304 can be connected via a bus or other means.

[0285] The computer-readable storage medium 1304 can be stored in the memory of a computer device. The computer-readable storage medium 1304 is used to store computer programs, including computer instructions. The processor 1301 is used to execute the program instructions stored in the computer-readable storage medium 1304. The processor 1301 (or CPU (Central Processing Unit)) is the computing and control core of the computer device, suitable for implementing one or more computer instructions, specifically suitable for loading and executing one or more computer instructions to achieve corresponding method flows or corresponding functions.

[0286] This application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the computer device. Furthermore, the storage space also stores one or more computer instructions suitable for loading and execution by a processor. These computer instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0287] In some embodiments, the processor 1301 may load and execute one or more computer instructions stored in the computer-readable storage medium 1304 to implement the aforementioned related... Figure 4 or Figure 5 The corresponding steps of the data processing method shown. In specific implementation, the computer instructions in the computer-readable storage medium 1304 are loaded by the processor 1301 and executed as follows:

[0288] The business processing model is invoked to perform business processing on the sample data; the business processing model includes an expert sub-model, which includes multiple expert networks, and the sample data includes multiple sample units.

[0289] During business processing, the expert allocation information of each sample unit in the sample data in the expert sub-model is obtained; the expert allocation information includes the probability information of the sample unit being assigned to each expert network in the expert sub-model for processing.

[0290] Based on the expert allocation information of each sample unit in the expert sub-model, the expert load balancing loss of the business processing model is determined. The expert load balancing loss of the business processing model is used to characterize the load balancing degree of each expert network in the expert sub-model when each sample unit in the sample data is allocated to the expert network in the expert sub-model for processing according to the corresponding probability information.

[0291] The business processing model is trained according to the rule of reducing the expert load balancing loss of the business processing model; the trained business processing model is used to identify and understand the content expressed by the media data.

[0292] In one implementation, when the computer instructions in the computer-readable storage medium 1304 are loaded and executed by the processor 1301 to determine the expert load balancing loss of the business processing model based on the expert allocation information of each sample unit in the expert sub-model, the instructions specifically perform the following steps:

[0293] The information content of expert allocation information for each sample unit in the expert sub-model is calculated to obtain the information content of expert allocation for each sample unit in the expert sub-model.

[0294] Based on the difference between the amount of expert-assigned information in each sample unit in the expert sub-model and the amount of information equally assigned in the expert sub-model, the expert load balancing loss of the business processing model is determined.

[0295] The information content of equal distribution in the expert sub-model refers to the amount of information when sample units are distributed with equal probability to each expert network in the expert sub-model for processing.

[0296] In one implementation, the business processing model includes multiple expert sub-models, and the target expert sub-model is any one of the expert sub-models included in the business processing model. When the computer instructions in the computer-readable storage medium 1304 are loaded and executed by the processor 1301 to determine the expert load balancing loss of the business processing model based on the difference between the amount of expert allocation information of each sample unit in the expert sub-model and the amount of equal allocation information of the expert sub-model, the following steps are specifically performed:

[0297] Based on the difference between the amount of information allocated by experts to each sample unit in the target expert sub-model and the amount of information allocated equally in the target expert sub-model, calculate the information content difference of each sample unit in the target expert sub-model.

[0298] The average of the information content difference of each sample unit in the target expert sub-model is calculated to obtain the expert load balancing loss of the target expert sub-model.

[0299] The expert load balancing loss of the business processing model is obtained by summing the expert load balancing losses of each expert sub-model included in the business processing model.

[0300] In one implementation, the target sample unit is any sample unit in the sample data; the expert sub-model also includes a gating network; the computer instructions in the computer-readable storage medium 1304 are loaded and executed by the processor 1301 during the business processing, when obtaining the expert allocation information of each sample unit in the sample data in the expert sub-model, specifically used to perform the following steps:

[0301] Obtain the vector representation of the target sample unit;

[0302] By invoking the gating network and performing expert network allocation processing on the target sample unit based on its vector representation, the expert allocation information of the target sample unit in the expert sub-model is obtained.

[0303] In one implementation, the sample data includes one or more training samples, each training sample including multiple sample units, and the target sample unit is any one of the sample units in the sample data; when the computer instructions in the computer-readable storage medium 1304 are loaded and executed by the processor 1301 to obtain the vector representation of the target sample unit, they are specifically used to perform the following steps:

[0304] Obtain the intermediate vector matrix corresponding to the target training sample to which the target sample unit belongs; the intermediate vector matrix is ​​composed of the intermediate vectors of the sample units included in the target training sample;

[0305] The intermediate vector matrix is ​​subjected to attention aggregation processing to obtain the attention vector matrix; the attention vector matrix includes the attention vectors of the sample units in the target training samples;

[0306] The attention vector of the target sample unit in the attention vector matrix is ​​determined as the vector representation of the target sample unit.

[0307] In one implementation, when the computer instructions in the computer-readable storage medium 1304 are loaded and executed by the processor 1301 to perform attention aggregation processing on the intermediate vector matrix to obtain the attention vector matrix, the following steps are specifically performed:

[0308] Based on the attention mechanism, h attention operations are performed on the intermediate vector matrix to obtain the attention head information corresponding to each attention operation in the h attention operations, where h is a positive integer;

[0309] The attention head information corresponding to each attention operation in h attention operations is concatenated to obtain the attention vector matrix.

[0310] In one implementation, the i-th attention operation is any one of the h attention operations, where i is a positive integer less than or equal to h; when the computer instructions in the computer-readable storage medium 1304 are loaded and executed by the processor 1301 to perform h attention operations on the intermediate vector matrix based on the attention mechanism to obtain the attention head information corresponding to each of the h attention operations, the instructions are specifically used to perform the following steps:

[0311] The intermediate vector matrix is ​​linearly transformed based on the linear transformation matrix corresponding to the i-th attention operation to obtain a linear vector matrix.

[0312] Normalize the linear vector matrix to obtain the attention head information corresponding to the i-th attention operation.

[0313] In one implementation, the computer instructions in the computer-readable storage medium 1304 are loaded by the processor 1301 and are also used to perform the following steps: during business processing, obtaining the business processing loss of the business processing model on the sample data;

[0314] The computer instructions in the computer-readable storage medium 1304, when loaded and executed by the processor 1301 to train the business processing model according to the rules for reducing the expert load balancing loss of the business processing model, are specifically used to perform the following steps:

[0315] The loss information of the business processing model is obtained by summing the business processing loss and the expert load balancing loss of the business processing model.

[0316] The business processing model is trained according to the rules for reducing the loss information of the business processing model.

[0317] In one implementation, the business processing model includes an encoding model and a decoding model; the sample data includes input text samples and output text samples, wherein the input text samples include multiple input sample units and the output text samples include multiple output sample units; the computer instructions in the computer-readable storage medium 1304 are loaded and executed by the processor 1301 during the business processing, and when obtaining the business processing loss of the sample data by the business processing model, they are specifically used to perform the following steps:

[0318] The encoding model is invoked to encode each input sample unit in the input text sample, and the encoding result of each input sample unit in the input text sample is obtained.

[0319] The decoding model is invoked to combine the encoding results of each input sample unit to perform word prediction on each output sample unit in the output text sample, thereby obtaining the probability distribution information of each output sample unit in the labeled vocabulary. The probability distribution information of the output sample unit in the labeled vocabulary includes the probability that the output sample unit is predicted as each word in the labeled vocabulary.

[0320] The business processing loss is calculated based on the probability distribution information of each output sample unit in the labeled vocabulary and the labeled position vector of each output sample unit in the labeled vocabulary.

[0321] In one implementation, the encoding model includes an expert sub-model; the reference input sample unit is any input sample unit in the input text sample; when the computer instructions in the computer-readable storage medium 1304 are loaded and executed by the processor 1301 to call the encoding model to encode each input sample unit in the input text sample and obtain the encoding result of each input sample unit in the input text sample, the following steps are specifically performed:

[0322] Obtain the processing results of the expert sub-model on the reference input sample unit;

[0323] The residual standardization process is applied to the processing results of the expert sub-model on the reference input sample unit to obtain the encoding result of the reference input sample unit.

[0324] In one implementation, when the computer instructions in the computer-readable storage medium 1304 are loaded and executed by the processor 1301 to obtain the processing results of the expert sub-model on the reference input sample unit, they are specifically used to perform the following steps:

[0325] Based on the expert assignment information of the reference input sample unit in the expert sub-model, determine one or more target expert networks in the expert sub-model for computing the reference input sample unit;

[0326] The target expert network is used to perform calculations on the reference input sample units, and the calculation results of the target expert network on the reference input sample units are obtained.

[0327] Based on the probability information of the reference input sample units being assigned to the target expert network for computation, the computation results of the target expert network on the reference input sample units are weighted and summed to obtain the processing results of the expert sub-model on the reference input sample units.

[0328] In one implementation, the computer instructions in the computer-readable storage medium 1304 are loaded by the processor 1301 and are also used to perform the following steps:

[0329] Obtain raw text samples for training the business processing model. The raw text samples include multiple sample units.

[0330] The input text sample is obtained by replacing some sample units in the original text sample with replacement units;

[0331] The replaced sample units are concatenated using replacement units to obtain the output text sample;

[0332] The input text samples and output text samples are combined to obtain training samples; the sample data includes one or more training samples.

[0333] In one implementation, the computer instructions in the computer-readable storage medium 1304 are loaded by the processor 1301 and are also used to perform the following steps:

[0334] Acquire media data to be processed, including any of the following: text, images, video, and audio;

[0335] The trained business processing model is invoked to perform content recognition on the media data, and the corresponding content tags for the media data are obtained.

[0336] In this application embodiment, an expert load balancing loss is proposed that can be used to characterize the load balancing degree of each expert network in the expert sub-model (i.e., MoE). In other words, the expert load balancing loss is directly related to the load balancing degree of each expert network in the expert sub-model. Therefore, by training the business processing model containing the expert sub-model with the expert load balancing loss, the load imbalance problem of each expert network in the expert sub-model included in the business processing model can be effectively improved.

[0337] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing methods provided in the various alternative embodiments described above.

[0338] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / y> < / x> < / y> < / x> < / y> < / x>

Claims

1. A data processing method, characterized in that, The method includes: The business processing model is invoked to perform business processing on the sample data; the business processing model includes an expert sub-model, which includes multiple expert networks; the sample data includes input text samples and output text samples, which include multiple input sample units and multiple output sample units. During the business processing, the expert allocation information of each sample unit in the sample data in the expert sub-model is obtained; the expert allocation information includes the probability information that the sample unit is assigned to each expert network in the expert sub-model for processing; the expert allocation information is used to determine one or more target expert networks for computing the sample unit in the expert sub-model. The information content of expert allocation information for each sample unit in the expert sub-model is calculated to obtain the expert allocation information content of each sample unit in the expert sub-model. Based on the difference between the expert allocation information content of each sample unit in the expert sub-model and the equal allocation information content of the expert sub-model, the expert load balancing loss of the business processing model is determined. The equal allocation information content of the expert sub-model refers to the information content when the sample units are allocated to each expert network in the expert sub-model with equal probability for processing. The expert load balancing loss characterizes the degree of load balancing among the expert networks in the expert sub-model when each sample unit in the sample data is allocated to the expert network in the expert sub-model for processing according to its corresponding probability information. The business processing model is trained according to the rules for reducing the expert load balancing loss of the business processing model; the trained business processing model is used to identify and understand the content expressed by the media data.

2. The method as described in claim 1, characterized in that, The business processing model includes multiple expert sub-models, and the target expert sub-model is any one of the expert sub-models included in the business processing model; determining the expert load balancing loss of the business processing model based on the difference between the amount of expert allocation information in each sample unit in the expert sub-model and the amount of information allocated equally in the expert sub-model includes: Based on the difference between the amount of expert-assigned information of each sample unit in the target expert sub-model and the amount of equally assigned information in the target expert sub-model, calculate the information content difference of each sample unit in the target expert sub-model. The average value of the information content difference of each sample unit in the target expert sub-model is calculated to obtain the expert load balancing loss of the target expert sub-model. The expert load balancing loss of the business processing model is obtained by summing the expert load balancing losses of each expert sub-model included in the business processing model.

3. The method as described in claim 1, characterized in that, The target sample unit is any sample unit in the sample data; the expert sub-model also includes a gating network; during the business processing, obtaining the expert allocation information of each sample unit in the sample data in the expert sub-model includes: Obtain the vector representation of the target sample unit; The gating network is invoked, and based on the vector representation of the target sample unit, expert network allocation processing is performed on the target sample unit to obtain the expert allocation information of the target sample unit in the expert sub-model.

4. The method as described in claim 3, characterized in that, The sample data includes one or more training samples, each training sample includes multiple sample units, and the target sample unit is any sample unit in the sample data; obtaining the vector representation of the target sample unit includes: Obtain the intermediate vector matrix corresponding to the target training sample to which the target sample unit belongs; the intermediate vector matrix is ​​composed of the intermediate vectors of the sample units included in the target training sample; The intermediate vector matrix is ​​subjected to attention aggregation processing to obtain an attention vector matrix; the attention vector matrix includes the attention vectors of the sample units in the target training sample; The attention vector of the target sample unit in the attention vector matrix is ​​determined as the vector representation of the target sample unit.

5. The method as described in claim 4, characterized in that, The attention aggregation process performed on the intermediate vector matrix to obtain the attention vector matrix includes: Based on the attention mechanism, h attention operations are performed on the intermediate vector matrix to obtain the attention head information corresponding to each attention operation in the h attention operations, where h is a positive integer; The attention head information corresponding to each attention operation in the h attention operations is concatenated to obtain the attention vector matrix.

6. The method as described in claim 5, characterized in that, The i-th attention operation is any one of the h attention operations, where i is a positive integer less than or equal to h; the h attention operations performed on the intermediate vector matrix based on the attention mechanism to obtain the attention head information corresponding to each of the h attention operations includes: The intermediate vector matrix is ​​linearly transformed according to the linear transformation matrix corresponding to the i-th attention operation to obtain a linear vector matrix; The linear vector matrix is ​​normalized to obtain the attention head information corresponding to the i-th attention operation.

7. The method as described in claim 1, characterized in that, The method further includes: During the business processing, the business processing loss of the sample data by the business processing model is obtained; The step of training the business processing model according to the rules for reducing the expert load balancing loss of the business processing model includes: The loss information of the business processing model is obtained by summing the business processing loss and the expert load balancing loss of the business processing model. The business processing model is trained according to the rules for reducing the loss information of the business processing model.

8. The method as described in claim 7, characterized in that, The business processing model includes an encoding model and a decoding model; the step of obtaining the business processing loss of the business processing model on the sample data during the business processing includes: The encoding model is invoked to encode each input sample unit in the input text sample, thereby obtaining the encoding result of each input sample unit in the input text sample; The decoding model is invoked to combine the encoding results of each input sample unit to perform word prediction on each output sample unit in the output text sample, thereby obtaining the probability distribution information of each output sample unit in the labeled vocabulary; the probability distribution information of the output sample unit in the labeled vocabulary includes: the probability that the output sample unit is predicted to be a word in the labeled vocabulary; The business processing loss is calculated based on the probability distribution information of each output sample unit in the labeled vocabulary and the labeled position vector of each output sample unit in the labeled vocabulary.

9. The method as described in claim 8, characterized in that, The encoding model includes the expert sub-model; the reference input sample unit is any input sample unit in the input text sample; the step of calling the encoding model to encode each input sample unit in the input text sample to obtain the encoding result of each input sample unit in the input text sample includes: Obtain the processing result of the expert sub-model on the reference input sample unit; The processing result of the expert sub-model on the reference input sample unit is subjected to residual standardization to obtain the encoding result of the reference input sample unit.

10. The method as described in claim 9, characterized in that, The step of obtaining the processing result of the expert sub-model on the reference input sample unit includes: Based on the expert allocation information of the reference input sample unit in the expert sub-model, one or more target expert networks are determined in the expert sub-model for computation of the reference input sample unit; The target expert network is used to perform calculations on the reference input sample unit to obtain the calculation results of the target expert network on the reference input sample unit; Based on the probability information of the reference input sample unit being assigned to the target expert network for computation, the computation results of the target expert network on the reference input sample unit are weighted and summed to obtain the processing result of the expert sub-model on the reference input sample unit.

11. The method as described in claim 1, characterized in that, The method further includes: Obtain raw text samples for training the business processing model, wherein the raw text samples include multiple sample units; The input text sample is obtained by replacing some sample units in the original text sample with replacement units; The replaced sample units are concatenated using the replacement units to obtain the output text sample; The input text sample and the output text sample are combined to obtain training samples; the sample data includes one or more of the training samples.

12. The method as described in claim 1, characterized in that, The method further includes: Acquire the media data to be processed, the media data including any of the following: text, images, video, and audio; The trained business processing model is invoked to perform content recognition on the media data, thereby obtaining the content tags corresponding to the media data.

13. A data processing apparatus, characterized in that, The device includes: The processing unit is used to call the business processing model to perform business processing on the sample data; the business processing model includes an expert sub-model, the expert sub-model includes multiple expert networks, the sample data includes input text samples and output text samples, the input text samples include multiple input sample units, and the output text samples include multiple output sample units; An acquisition unit is configured to acquire, during the business processing, expert allocation information of each sample unit in the sample data within the expert sub-model; the expert allocation information includes probability information of the sample unit being assigned to each expert network in the expert sub-model for processing; the expert allocation information is used to determine one or more target expert networks in the expert sub-model for computation of the sample unit. The processing unit is further configured to calculate the information content of the expert allocation information of each sample unit in the expert sub-model, thereby obtaining the expert allocation information content of each sample unit in the expert sub-model; and to determine the expert load balancing loss of the business processing model based on the difference between the expert allocation information content of each sample unit in the expert sub-model and the equal allocation information content of the expert sub-model; wherein, the equal allocation information content of the expert sub-model refers to the information content when the sample units are allocated to each expert network in the expert sub-model with equal probability for processing; and the expert load balancing loss is used to characterize the load balancing degree of each expert network in the expert sub-model when each sample unit in the sample data is allocated to the expert network in the expert sub-model for processing according to the corresponding probability information. The processing unit is also used to train the business processing model according to the rules for reducing the expert load balancing loss of the business processing model; the trained business processing model is used to identify and understand the content expressed by the media data.

14. A computer device, characterized in that, The computer device includes: A processor is a tool for implementing computer programs. A computer-readable storage medium storing a computer program adapted to be loaded by the processor and executed as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, implement the data processing method as described in any one of claims 1 to 12.

Images