Label classification method and device, label classification model construction method and device, equipment and medium
By splitting the pre-trained large language model into a hybrid model of expert network and routing layer, the label classification efficiency problem under the massive label system is solved, efficient and accurate label classification is achieved, and training and iterative costs are reduced.
Patent Information
- Application Number
- CN202510225727.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-27
- Publication Date
- 2025-07-22
AI Technical Summary
When faced with massive label systems, the existing technology has problems such as low tag classification efficiency, long training cycle, poor model generalization performance, and long iteration cycle of the label system.
The pre-trained large language model is transformed into a hybrid expert model. By splitting the feedforward neural network into multiple expert networks, and building a routing layer, optimizing the label classification model, and using the strong generalization ability and sparseness of the large language model to improve processing efficiency.
On the basis of maintaining the performance of the original model, the processing efficiency of the tag classification processing task is enhanced, efficient and accurate classification of massive tags is achieved, and training costs and iteration cycles are reduced.
Smart Images
Figure CN120353930A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of natural language processing, and particularly to label classification and its model construction methods, devices, equipment, and media. Background Art
[0002] In the Internet environment, content distribution platforms have accumulated a large amount of text data. To improve the user experience, it is necessary to perform high-quality characterization and distribution of content. Content tagging is an important link in content distribution. Usually, a modeling method based on a pre-trained model is used to perform the tagging task. This method is mainly used in scenarios with a small number of labels and a fixed label system, and the modeling method is relatively simple. In the face of a large number of label systems, there are problems such as a long training cycle, poor model generalization performance, and a long label system iteration cycle. Therefore, in the face of a large amount of content and labels, how to perform high-quality tagging is an urgent problem to be solved currently. Summary of the Invention
[0003] This application provides a label classification and its model construction method, device, equipment, and media to at least solve the problem of low label classification efficiency in related technologies.
[0004] This application provides a method for constructing a label classification model, including:
[0005] Obtain a first language model, and based on the weight matrix of the feed-forward neural network of the first language model, split the feed-forward neural network to determine multiple expert networks;
[0006] Based on a preset routing layer and the expert networks, determine a first output result of the split feed-forward neural network;
[0007] Based on the first output result and the second output result corresponding to the first language model, optimize the label classification model, and the label classification model is at least composed of the expert networks obtained by splitting the feed-forward neural network of the first language model.
[0008] This application provides a label classification method, including:
[0009] Obtain the label classification model constructed based on the above label classification model construction method, and deploy the label classification model;
[0010] Obtain a target text;
[0011] Based on the label classification model, process the target text to determine the tagging result of the target text, and the tagging result at least includes the probability distribution of the target text labels.
[0012] This application also provides a label classification model construction device, including:
[0013] A network splitting module, configured to obtain a first language model, and based on the weight matrix of the feed-forward neural network of the first language model, split the feed-forward neural network to determine a plurality of expert networks;
[0014] A first result determination module, configured to determine a first output result of the split feed-forward neural network based on a preset routing layer and the expert networks;
[0015] A model optimization module, configured to optimize a label classification model based on the first output result and a second output result corresponding to the first language model, where the label classification model is at least composed of expert networks obtained by splitting the feed-forward neural network of the first language model.
[0016] This application also provides a computer device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any of the above-mentioned label classification model construction methods and / or label classification methods when executing the computer program.
[0017] This application also provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above-mentioned label classification model construction methods and / or label classification methods are implemented.
[0018] Through this application, the provided label classification model construction method transforms the first language model into a model with expert networks. Since the first language model has strong text understanding ability after pre-training, using the sparsity of activation in the feed-forward neural network, the feed-forward neural network is split into independent parts as the expert networks of the label classification model, which can enhance the processing efficiency of the label classification task while maintaining the performance of the original model. Description of the Drawings
[0019] To more clearly illustrate the embodiments of this application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0020] Figure 1 It is a flowchart of a label classification model construction method provided by an embodiment of this application;
[0021] Figure 2 It is a schematic diagram of the model fine-tuning process provided by an embodiment of this application;
[0022] Figure 3 It is a schematic diagram of the computer device provided by an embodiment of this application. Detailed Description of the Embodiments
[0023] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.
[0024] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variation thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0025] In order to enable those skilled in the art of this technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0026] In the current Internet environment, content distribution platforms have accumulated a vast amount of text data. To improve the user experience, it is necessary to characterize and distribute this text data. Content tagging is one of the important links in content distribution. As an important part of content understanding, content tagging plays a crucial role in achieving fine-grained understanding and operation of content, as well as accurately characterizing user interests. Traditional methods are based on pre-trained language models, which are fine-tuned on specific datasets to obtain trained language models. Suppose the task involves K categories. For each input sample, which is generally the title of an article, after passing through the pre-trained language model, a feature vector is obtained. To calculate the loss of the sample, a multi-classification loss function, such as multi-classification cross-entropy, is usually used to measure the difference between the predicted label and the true label. Finally, the pre-trained output of the model is connected to a fully connected layer, and the output value is mapped to between 0 and 1 through an activation function. Each element of this vector represents the probability that the sample belongs to the corresponding label. Based on this probability, the label of the article can be determined, thus achieving multi-label classification of text in a specific scenario. However, this method has problems such as a long training cycle and poor model generalization ability when dealing with a large number of labels.
[0027] Based on this, the embodiments of the present invention provide a method for label classification and its model construction, which is used to construct a label classification model and perform label classification based on the constructed model. The method for constructing the label classification model transforms a pre-trained large language model into a mixture of experts model. By utilizing the sparsity activation in the fully connected layer (FFN, Feed Forward Network) of the model, the FFN is split into several independent parts and converted into the expert layer part of the label classification model. Then, a router is trained to select the appropriate expert for each input. The training objective is to ensure that the output of the FFN before and after splitting is as consistent as possible, thereby maintaining most of the performance of the original model. The training of the router can be carried out simultaneously with the fine-tuning of a large number of label classification tasks to reduce the training cost. This method utilizes the advantages of high accuracy and strong generalization ability of the large language model to achieve efficient and accurate classification of a large number of labels.
[0028] A pre-trained language model generally refers to designing a language model training task based on a large-scale corpus (including language training materials such as sentences, paragraphs, etc.), training a large-scale neural network algorithm structure to learn and implement. The finally obtained large-scale neural network algorithm structure and parameters are the pre-trained language model. For subsequent other tasks, feature extraction or task fine-tuning can be performed on the basis of this model to achieve specific task purposes. The idea of pre-training is to first train a task to obtain a set of model parameters, then use this set of model parameters to initialize the network model parameters, and then use the initialized network model to train other tasks to obtain models adapted to other tasks. By pre-training on a large-scale corpus, the neural language representation model can learn powerful language representation capabilities and extract rich syntactic and semantic information from the text. The pre-trained language model can provide word elements (tokens) containing rich semantic information and sentence-level features for downstream tasks, or directly perform fine-tuning for downstream tasks on the pre-trained model to conveniently and quickly obtain a downstream-specific model.
[0029] The neural network algorithm structure trained by the pre-trained language model can be CNN, RNN, LSTM, etc., or a model constructed with an attention network, such as Transformer, bert, GPT, Clip, etc. This application does not make any limitations here. An attention network refers to a network model trained using the attention mechanism. This model assigns different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence and enabling the model to finally obtain a more accurate output.
[0030] The FFN layer (Feed-Forward Network layer) is a key component in the Transformer architecture (a deep learning model). Essentially, it is a simple multi-layer perceptron (MLP) used for non-linear transformation of features in the Transformer model. It usually consists of two linear transformations (i.e., fully connected layers) sandwiching an activation function to introduce non-linearity. The attention layer is mainly responsible for learning the relationships between different tokens, while the FFN layer processes each token individually, which helps the model capture local and non-linear features, thereby increasing the model's expressive power. Through high-dimensional transformation and activation functions, the FFN can generate more complex feature representations for each token, which helps the model capture more useful information when processing sequence data.
[0031] Fine-tuning refers to performing small-scale training on a pre-trained model for specific task objectives (downstream tasks) and task data (downstream data), achieving minor adjustments to the parameters of the pre-trained model, and ultimately obtaining a model adapted to specific tasks and data.
[0032] In this embodiment, a method for constructing a label classification model is provided. Figure 1 It is a flowchart of the method for constructing a label classification model according to an embodiment of the present invention, as Figure 1 shown. The process includes the following steps:
[0033] Step S101, obtain a first language model, and split the feed-forward neural network based on the weight matrix of the feed-forward neural network of the first language model to determine multiple expert networks.
[0034] The first language model is a pre-trained large language model. The first language model has been pre-trained on a large amount of text data and has language understanding and generation capabilities. This model has a deep learning model architecture (e.g., Transformer architecture) and its related structures, and has a feed-forward neural network layer. The feed-forward neural network is used in the first language model to process the output after the self-attention layer in the Transformer. Split the feed-forward neural network layer into multiple smaller networks, and each network is an expert network. An expert network is a set of neurons that are activated simultaneously and can work in parallel to process a subset of the input data or specific types of features.
[0035] The splitting is based on the weight matrix of the feed-forward neural network. Split the weight matrix into multiple sub-matrices, and each sub-matrix corresponds to an expert network. Each expert network has a corresponding weight set for processing the input data. The number of expert networks can be set according to actual needs.
[0036] During splitting, the sparsity of activation in the feedforward neural network layer is utilized, that is, most inputs only activate a small number of neurons. Neurons that are often activated simultaneously are grouped together and regarded as an expert model. In this way, only a small amount of expert calculations can be used to achieve an output result similar to that of the original feedforward neural network layer.
[0037] Step S102: Based on the preset routing layer and the expert network, determine the first output result of the split feedforward neural network.
[0038] After splitting the feedforward neural network, a routing layer (router) is constructed to decide which or which expert networks should be used for each input. The routing layer is a lightweight neural network that can generate a probability distribution based on the characteristics of the input data, and this distribution represents the probability of each expert network being selected.
[0039] Once the routing layer decides which expert networks should process each input data, the expert networks will process the input data in parallel and generate their respective outputs, which will then be merged to obtain the first output result of the split feedforward neural network.
[0040] Step S103: Optimize the label classification model based on the first output result and the second output result corresponding to the first language model.
[0041] Among them, the label classification model is at least composed of the expert networks obtained after splitting the feedforward neural network of the first language model.
[0042] After splitting the feedforward neural network of the first language model, an expert network model with an untrained routing layer is obtained. In this solution, this model is the label classification model used to perform the label classification task. Therefore, it is necessary to train the routing layer of the label classification model. In addition, in order to adapt to the downstream task of massive label classification, it is necessary to fine-tune the model on the label classification task dataset.
[0043] The second output result is the processing result of the original first language model on the input data before splitting, which is the same as the input data of the label classification model containing the expert network obtained after splitting. That is, for the same input data, the output result of the label classification model is the first output result, and the output result of the first language model is the second output result. The input data used is the label classification dataset for training.
[0044] The first output result and the second output result are feature vectors based on a label classification model and a first language model respectively, and loss calculation is performed according to the difference between the first output result and the second output result. Additionally, loss calculation is performed using the feature vector obtained from the label classification model and the classification label of the input data (label classification dataset), and this loss is the loss during normal downstream fine-tuning tasks such as cross-entropy loss. The calculated losses are combined (calculate the sum of the losses) to obtain the total loss. The total loss is optimized to fine-tune the label classification model.
[0045] After fine-tuning the label classification model, the performance of the label classification model can be evaluated on the validation set to ensure that the model meets the required standards.
[0046] Exemplarily, the dataset used in the model fine-tuning process is text data, and each piece of data contains text content and labels (which can be single or multiple).
[0047] For example: {'text': 'Recently, various migratory birds have been observed in a certain community park, and residents have spontaneously organized ecological observation activities and made science popularization brochures', 'label1': 'Environmental Protection', 'label2': 'Nature Science Popularization'}.
[0048] This example is in JSON format, and the format of the input data is not limited in actual applications.
[0049] The method for constructing a label classification model provided in this embodiment includes obtaining a first language model, splitting the feedforward neural network based on the weight matrix of the feedforward neural network of the first language model to determine multiple expert networks; determining the first output result of the split feedforward neural network based on a preset routing layer and the expert networks; and optimizing the label classification model based on the first output result and the second output result corresponding to the first language model. The method for constructing a label classification model provided by the present invention transforms the first language model into a model with expert networks. Since the first language model has strong text understanding ability after pre-training, using the sparsity of activation in the feedforward neural network, the feedforward neural network is split into independent parts as the expert networks of the label classification model, which can enhance the processing efficiency of the label classification task while maintaining the performance of the original model.
[0050] Optionally, different types of expert networks can be designed, and different expert networks are used to process different feature patterns. When selecting an expert network, the most suitable one or more expert networks can be selected according to the input feature type.
[0051] Optionally, when optimizing the model, new expert networks can be dynamically added to adapt to new tasks or new data distributions. The new expert networks can initialize their parameters from existing experts through transfer learning to accelerate convergence. This can improve the adaptability of the model in continuous learning scenarios and avoid retraining the entire model, reducing computational costs.
[0052] In this embodiment, a method for constructing a label classification model is provided. Figure 1 It is a flowchart of the method for constructing a label classification model according to an embodiment of the present invention, as Figure 1 shown. The process includes the following steps:
[0053] Step S201: Obtain a first language model, and split the feed-forward neural network based on the weight matrix of the feed-forward neural network of the first language model to determine multiple expert networks.
[0054] Specifically, step S201 includes the following steps:
[0055] Step S2011: Perform clustering on the vector set in the weight matrix of the first language model to obtain a permutation matrix.
[0056] Step S2012: Based on a preset number of experts and the permutation matrix, rearrange the weight matrix of the first language model to determine the input weights and output weights of the expert networks.
[0057] Each column vector of the feed-forward neural network of the first language model is regarded as a vector with dimension d, and these column vectors form a vector set. In the computational structure of the feed-forward neural network, the weight matrix can map the input to the intermediate layer, and its column vectors correspond to different neurons, so these column vectors can represent the characteristics of the neurons.
[0058] The purpose of this step is to rearrange the weight matrix of the feed-forward neural network layer according to the neuron activation pattern to group and form expert networks. Based on the clustering method, neurons that are frequently co-activated in the feed-forward neural network layer are grouped, and a permutation matrix is generated according to the clustering result. The role of the permutation matrix is to rearrange the columns of the feed-forward neural network of the first language model so that the columns corresponding to neurons belonging to the same expert are adjacent.
[0059] The K-means clustering algorithm can be used to continue clustering the vector set. The K-means clustering algorithm is an unsupervised learning algorithm, and its goal is to divide data points into k different clusters so that data points within the same cluster are as similar as possible, and data points between different clusters are as different as possible. In this embodiment, similar neurons are grouped into the same cluster, and each cluster corresponds to an expert network.
[0060] Further, the weight matrix of the first language model includes a first weight matrix and a second weight matrix, and the input weights and output weights of the expert network are determined according to the following formula:
[0061]
[0062] where W1 represents the first weight matrix, W2 represents the second weight matrix, P represents the permutation matrix, k represents the preset number of experts, and the sub-matrix represents the input weight of the k-th expert network, and the sub-matrix represents the output weight of the k-th expert network.
[0063] Rearranging the columns of the first weight matrix W1 gives k sub-matrices Each sub-matrix represents the input weight of the k-th expert network; rearranging the second weight matrix W2 gives k sub-matrices Each sub-matrix represents the output weight of the k-th expert network.
[0064] Dividing the parameters of the feedforward neural network into multiple parts according to the above splitting results, each part becomes an expert network. The input and output dimensions of each expert network are still d, and the intermediate dimension is d / k. When each expert processes the input, it first maps the input through the input weight of the expert network to the intermediate layer with dimension d / k, and then maps the output of the intermediate layer back to the output space with dimension d through the output weight The permutation matrix P is a permutation matrix that reorders the columns of the first weight matrix and the second weight matrix according to the clustering result, thereby splitting and matching the parameters of the feedforward neural network to each expert network.
[0065] The splitting step of the feedforward neural network utilizes the sparsity of activation in the feedforward neural network layer, that is, most inputs only activate a small number of neurons. In the calculation of the feedforward neural network, neurons that are often activated simultaneously can be grouped together and regarded as an expert model, so that the output result similar to that of the original feedforward neural network layer can be achieved with only a small number of expert calculations.
[0066]
[0067] Exemplarily, assume that the first weight matrix W1 of the original first language model is a 1024*2048 matrix, and the preset number of experts is 8. Consider the 1024*2048 matrix as 1024 vectors of length 2048. Use a clustering algorithm to divide these vectors into 8 groups, and change the arrangement of these vectors according to the grouping results. What is obtained in this step is W1P, and W1P is also a 1024*2048 matrix, and the order of the rows is different from that of the first weight matrix W1.
[0068] Let k = 8. Here, the 1024*2048 matrix is split into 8 1024*256 matrices. This is equivalent to dividing the second dimension 2048 by 8, and each is a 1024*256 sub-block of the original matrix.
[0069] For the second weight matrix W2, it is the first dimension that is split. Assume that W2 is a 2048*1024 matrix, then split the 2048 dimension and turn it into 8 256*1024 matrices.
[0070] Step S202: Based on the preset routing layer and expert networks, determine the first output result of the feedforward neural network after splitting.
[0071] Specifically, step S202 includes the following steps:
[0072] Step S2021: Obtain training data, use the training data as the input of the preset routing layer, and obtain an output tensor.
[0073] Construct a routing layer (router) to determine which experts should be used for each input. The purpose is to select the most suitable subset of experts for each input to approximate the output of the original feedforward neural network layer. After splitting the feedforward neural network of the first language model, train a linear layer as the routing layer. The routing layer outputs the probability that each expert network is selected. The input dimension is d, and the output dimension is k. The training objective is to minimize the difference between the output results of the feedforward neural network before and after splitting. Input the training data into a label classification model containing expert networks. The training data passes through the preset routing layer and outputs a tensor of length k.
[0074] Step S2022: Based on the normalization processing result of the output tensor, determine the set of expert networks.
[0075] Perform normalization processing on the output tensor. After normalization, the expert networks to be activated can be selected according to the magnitude of the weights. For example, select the first few expert networks with the largest weights, or select expert networks according to a certain strategy (such as sparsity requirements). Exemplarily, select the first n largest terms to form the set of expert networks.
[0076] Step S2023, based on the expert network ensemble, activation function, input weights of the expert network, and output weights of the expert network, calculate the first output result of the split feedforward neural network.
[0077] Specifically, the first output result is determined according to the following formula:
[0078]
[0079] Among them, S represents the expert network ensemble, σ represents the activation function, represents the input weight of the i-th expert network, represents the output weight of the i-th expert network, p i represents the element of the preset routing layer, and F′(x) represents the first output result.
[0080] Step S203, optimize the label classification model based on the first output result and the second output result corresponding to the first language model.
[0081] Specifically, step S203 includes the following steps:
[0082] Step S2031, use the training data as the input of the first language model to obtain the second output result.
[0083] Step S2032, determine the first loss based on the difference between the first output result and the second output result.
[0084] Output the training data to the first language model to obtain the second output result output by the first language model. The first output result and the second output result are the feature vectors calculated by the label classification model and the first language model based on the training data respectively. Calculate the difference between the two feature vectors to obtain the first loss.
[0085] Specifically, the calculation method of the first difference Loss1 is as follows:
[0086] Loss1 = [F(x) - F′(x)] 2
[0087] Among them, Loss1 represents the first difference, F′(x) represents the first output result, and F(x) represents the second output result. By minimizing Loss1, the split model can be made as close as possible to the output before splitting, preserving the original understanding ability of the model.
[0088] Step S2033, calculate the loss based on the feature vector of the label classification model and the classification label of the label classification data set to obtain the second loss.
[0089] Step S2034: Determine the total loss based on the first loss and the second loss, and optimize the label classification model based on the total loss.
[0090] Use the label classification model to obtain the feature vector and the classification label of the label classification data set for loss calculation, and obtain the second loss Loss2. The second loss is the loss when performing normal downstream fine-tuning tasks such as cross-entropy loss.
[0091] The total loss is L:
[0092] L = Loss1 + Loss2
[0093] Optimize the total loss, thereby optimizing the label classification model.
[0094] To reduce costs, the first loss and the second loss can be calculated simultaneously.
[0095] As a specific implementation manner, the embodiment of the present application provides a method for training a label classification model. Let the first language model be a pre-trained large language model, and the first language model has a Transformer-related structure. Among them, Transformer is a deep learning model architecture, and the Transformer structure consists of two parts: an encoder and a decoder, and each part is composed of the same layer stack. The FFN layer (Feed Forward Network) is a component in the Transformer architecture, and the FFN layer includes a linear transformation and an activation function. The first linear transformation expands the dimension of the input vector to a higher dimension, then applies the activation function, and finally restores the dimension to the original size through the second linear transformation.
[0096] After obtaining the first language model, set the number of experts according to actual needs, so as to determine how many expert networks the FFN layer needs to be split into. When splitting the FFN, based on the sparsity of activation in the FFN layer, that is, most inputs only activate a small number of neurons. In the calculation of the FFN, neurons that are often activated simultaneously can be grouped together and regarded as an expert model, so that the output result similar to the original FFN layer can be achieved with only a small number of expert calculations. When constructing experts, cluster the column vectors of the FFN layer weight matrix into multiple groups, and construct a permutation matrix according to the clustering result, which is used to rearrange the columns or rows of the weight matrix by expert groups, so as to calculate the expert network.
[0097] After constructing the expert network, a routing layer is constructed. The routing layer can be used to determine which experts should be used for each input. The goal is to select the most appropriate subset of experts for each input to approximate the output of the original FFN layer. After splitting the above FFN layer, a linear layer is trained as a router. The router outputs the probability of each expert being selected. The input dimension is d and the output dimension is k. The training goal is to minimize the difference between the output results of the FFN before and after splitting.
[0098] After splitting, the model changes from a pre-trained transformer structure model to a MoE model (Mixture of Experts) with an untrained router, as Figure 2 shown in the schematic diagram of model fine-tuning process. On the one hand, the router is trained, that is, the first loss is calculated. On the other hand, in order to adapt to the downstream task of massive label classification, the model needs to be fine-tuned on the label classification task dataset, that is, the second loss is calculated.
[0099] The training goal of the router is to optimize the expert assignment strategy to ensure that each expert can handle appropriate tasks. The goal of fine-tuning is to improve the performance of the model on specific tasks. These two goals are complementary to each other during the training process and can be carried out simultaneously.
[0100] The label classification model construction method provided by the embodiments of the present invention transforms a pre-trained large language model into a mixture of experts model structure to handle massive text label classification tasks. This method combines the advantages of high accuracy and strong generalization ability of the large language model to achieve efficient and accurate classification of massive labels. This model has strong generalization ability. Since the large language model has been pre-trained on massive text data, it can learn rich language features and knowledge, thus showing higher accuracy in text classification tasks, stronger generalization ability, and being able to maintain good performance in different fields and scenarios. The iteration efficiency is high. When new labels need to be added according to requirements, due to the strong generalization ability of the large language model, adding a small number of labels or labels that are not completely new types does not require re-fine-tuning, ensuring the iteration efficiency of the label system. It has good inference efficiency. By transforming the pre-trained large language model into a MoE model, the inference efficiency of the model can be improved while maintaining the performance of the original model.
[0101] In this embodiment, a label classification method is provided, including: obtaining a label classification model constructed by the label classification model construction method based on the above embodiment, and deploying the label classification model; obtaining a target text; processing the target text based on the label classification model to determine the labeling result of the target text. The labeling result at least includes the probability distribution of the target text labels.
[0102] Deploy the label classification model constructed based on the above-mentioned label classification model construction method to an environment that can process requests, such as a server, cloud service, etc. After the model is deployed, obtain the target text to be classified, and the target text can come from various sources such as user input and file reading.
[0103] After performing certain preprocessing on the target text, input the target text into the label classification model. After the label classification model processes the target text, it outputs the probability distribution of one or more labels. According to this probability distribution, take the labels corresponding to the first few maximum probabilities as the labels of the target text.
[0104] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.
[0105] The embodiments of the present application also provide a label classification model construction device, including:
[0106] A network splitting module, configured to obtain a first language model, and based on the weight matrix of the feedforward neural network of the first language model, split the feedforward neural network to determine a plurality of expert networks;
[0107] A first result determination module, configured to determine a first output result of the split feedforward neural network based on a preset routing layer and the expert networks;
[0108] A model optimization module, configured to optimize the label classification model based on the first output result and the second output result corresponding to the first language model, and the label classification model is at least composed of the expert networks obtained by splitting the feedforward neural network of the first language model.
[0109] In some optional embodiments, the network splitting module includes:
[0110] A clustering processing unit, configured to perform clustering processing on the vector set in the weight matrix of the first language model to obtain a permutation matrix;
[0111] An expert network determination unit, configured to rearrange the weight matrix of the first language model based on a preset number of experts and the permutation matrix to determine the input weight and output weight of the expert network.
[0112] In some optional embodiments, the weight matrix of the first language model includes a first weight matrix and a second weight matrix, and the input weight and output weight of the expert network are determined according to the following formula:
[0113]
[0114] Among them, W1 represents the first weight matrix, W2 represents the second weight matrix, P represents the permutation matrix, k represents the preset number of experts, and the sub-matrix represents the input weight of the k-th expert network, and the sub-matrix represents the output weight of the k-th expert network.
[0115] In some alternative embodiments, the first result determination module includes:
[0116] An output tensor determination unit, configured to obtain training data, use the training data as the input of a preset routing layer, and obtain an output tensor;
[0117] A set determination unit, configured to determine a set of expert networks based on the normalization processing result of the output tensor;
[0118] A first output result determination unit, configured to calculate a first output result of the split feedforward neural network based on the set of expert networks, an activation function, the input weight of the expert network, and the output weight of the expert network.
[0119] In some alternative embodiments, the first output result is determined according to the following formula:
[0120]
[0121] where S represents the set of expert networks, σ represents the activation function, represents the input weight of the i-th expert network, represents the output weight of the i-th expert network, p i represents an element of the preset routing layer, and F ′ (x) represents the first output result.
[0122] In some alternative embodiments, the model optimization module includes:
[0123] A second output result determination unit, configured to use the training data as the input of the first language model to obtain a second output result;
[0124] A first loss determination unit, configured to determine a first loss based on the difference between the first output result and the second output result;
[0125] A second loss determination unit, configured to calculate a loss based on the feature vector of the label classification model and the classification label of the label classification data set to obtain a second loss;
[0126] A model optimization unit, configured to determine a total loss based on the first loss and the second loss, and optimize the label classification model based on the total loss.
[0127] For the description of the features in the corresponding embodiments of the label classification model construction device, reference can be made to the relevant descriptions in the corresponding embodiments of the label classification model construction method, which will not be elaborated here one by one.
[0128] An embodiment of the present application also provides a computer device, such as Figure 3 shown, including a processor 10 and a memory 20. A computer program is stored in the memory 20, and the processor 10 is configured to run the computer program to execute the steps of any of the above-mentioned label classification model construction methods and / or label classification methods.
[0129] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned label classification model construction methods and / or label classification methods when running.
[0130] In an exemplary embodiment, the above-mentioned computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs and other media that can store computer programs.
[0131] An embodiment of the present application also provides a computer program product. The above-mentioned computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned label classification model construction methods and / or label classification methods are implemented.
[0132] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned label classification model construction methods and / or label classification methods are implemented.
[0133] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.
[0134] The above has introduced the method provided by this application in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and modifications can still be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.
Claims
1. A method for constructing a label classification model, characterized in that including: obtaining a first language model, and splitting the feedforward neural network based on the weight matrix of the feedforward neural network of the first language model to determine a plurality of expert networks; determining a first output result of the split feedforward neural network based on a preset routing layer and the expert networks; optimizing a label classification model based on the first output result and a second output result corresponding to the first language model, where the label classification model is at least composed of expert networks obtained by splitting the feedforward neural network of the first language model.
2. The method for constructing a label classification model according to claim 1, wherein The splitting the feedforward neural network based on the weight matrix of the feedforward neural network of the first language model to determine a plurality of expert networks includes: performing clustering processing on a vector set in the weight matrix of the first language model to obtain a permutation matrix; rearranging the weight matrix of the first language model based on a preset number of experts and the permutation matrix to determine the input weights and output weights of the expert networks.
3. The method for constructing a label classification model according to claim 2, wherein The weight matrix of the first language model includes a first weight matrix and a second weight matrix, and the input weights and output weights of the expert networks are determined according to the following formula: Among them, W1 represents the first weight matrix, W2 represents the second weight matrix, P represents the permutation matrix, k represents the preset number of experts, and the sub-matrix represents the input weight of the k-th expert network, and the sub-matrix represents the output weight of the k-th expert network.
4. The method for constructing a label classification model according to claim 1, characterized in that The determining a first output result of the split feedforward neural network based on a preset routing layer and the expert networks includes: obtaining training data, using the training data as the input of the preset routing layer to obtain an output tensor; determining a set of expert networks based on the normalization processing result of the output tensor; calculating a first output result of the split feedforward neural network based on the set of expert networks, an activation function, the input weights of the expert networks, and the output weights of the expert networks.
5. The method for constructing a label classification model according to claim 4, characterized in that, The first output result is determined according to the following formula: Among them, S represents the set of expert networks, and σ represents the activation function. represents the input weight of the i-th expert network. represents the output weight of the i-th expert network, p i represents the element of the preset routing layer, F ′ (x) represents the first output result.
6. The method for constructing a label classification model according to claim 1, wherein The optimizing a label classification model based on the first output result and a second output result corresponding to the first language model includes: using the training data as the input of the first language model to obtain a second output result; determining a first loss based on the difference between the first output result and the second output result; calculating a second loss based on the feature vectors of the label classification model and the classification labels of the label classification data set; determining a total loss based on the first loss and the second loss, and optimizing the label classification model based on the total loss.
7. A method for classifying labels, characterized in that, including: obtaining a label classification model constructed by using the label classification model construction method according to any one of claims 1 to 6, and deploying the label classification model; obtaining a target text; processing the target text based on the label classification model to determine a tagging result of the target text, where the tagging result at least includes a probability distribution of the target text label.
8. A device for constructing a label classification model, characterized in that including: a network splitting module, configured to obtain a first language model, and split the feedforward neural network based on the weight matrix of the feedforward neural network of the first language model to determine a plurality of expert networks; a first result determining module, configured to determine a first output result of the split feedforward neural network based on a preset routing layer and the expert networks; A model optimization module, configured to optimize a label classification model based on the first output result and a second output result corresponding to the first language model, where the label classification model is at least composed of an expert network obtained by splitting a feedforward neural network of the first language model.
9. A computer device, characterized in that, It includes: A memory, configured to store a computer program; A processor, configured to implement the label classification model construction method according to any one of claims 1 to 6 and / or the label classification method according to claim 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program, when executed by a processor, implements the label classification model construction method according to any one of claims 1 to 6 and / or the label classification method according to claim 7.