Text classification method, and deep learning model training method and device

Abstract

The invention provides a text classification method and a deep learning model training method and device, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, natural language processing, content security and intelligent search. According to the specific implementation scheme, the method comprises the steps of generating semantic features of an input text by using a shared base network; according to the semantic features and a splicing weight matrix of the plurality of tower networks, determining respective classification results of a plurality of text classification tasks respectively corresponding to the plurality of tower networks, the splicing weight matrix being obtained by splicing weight matrixes of the same processing layer of the plurality of tower networks; and according to respective classification results of the plurality of text classification tasks, determining a target classification result of the input text.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the fields of deep learning, natural language processing, content security, and intelligent search technology. More specifically, this disclosure provides a text classification method, a deep learning model training method, an apparatus, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] With the continuous development of computer and internet technologies, information is growing explosively, and the demand for cybersecurity is increasing daily. For example, in search scenarios, identifying the risk intent of user-entered search queries is crucial for maintaining content security and ensuring user experience. Summary of the Invention

[0003] This disclosure provides a text classification method, a deep learning model training method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to the first aspect, a text classification method is provided, which includes: generating semantic features of input text using a shared pedestal network; determining the classification results of multiple text classification tasks corresponding to the multiple pedestal networks based on the semantic features and the concatenated weight matrix of the multiple pedestal networks, wherein the concatenated weight matrix is ​​obtained by concatenating the weight matrices of the same processing layer of the multiple pedestal networks; and determining the target classification result of the input text based on the classification results of the multiple text classification tasks.

[0005] According to the second aspect, a training method for a deep learning model is provided. The deep learning model includes a shared pedestal network and multiple tower networks corresponding to classification tasks for multiple vertical categories. The method includes: inputting sample data into the shared pedestal network to obtain semantic features of the sample data, wherein the sample data includes multiple sample texts with category labels for multiple vertical categories; inputting the semantic features into the multiple tower networks to obtain the classification result of each tower network for the sample text; calculating the loss of each tower network based on the category labels of the sample text and the classification result, and calculating the global loss of the multiple tower networks based on the mutual exclusion between the category labels of the multiple vertical categories; and adjusting the parameters of the tower networks based on the individual losses of the multiple tower networks and the global loss of the multiple tower networks.

[0006] According to a third aspect, a text classification device is provided, comprising: a first semantic feature determination module, used to generate semantic features of input text using a shared pedestal network; a first classification module, used to determine the classification results of multiple text classification tasks corresponding to the multiple pedestal networks respectively, based on the semantic features and a concatenated weight matrix of the multiple pedestal networks, wherein the concatenated weight matrix is ​​obtained by concatenating the weight matrices of the same processing layers of the multiple pedestal networks; and a classification result determination module, used to determine the target classification result of the input text based on the classification results of the multiple text classification tasks.

[0007] According to the fourth aspect, a training device for a deep learning model is provided. The deep learning model includes a shared pedestal network and multiple tower networks corresponding to classification tasks of multiple vertical categories. The device includes: a second semantic feature determination module, used to input sample text into the shared pedestal network to obtain semantic features of the sample text, wherein the sample text includes multiple sample texts with category labels of multiple vertical categories; a second classification module, used to input the semantic features into the multiple tower networks to obtain the classification result of each tower network for the sample text; a loss determination module, used to calculate the loss of each of the multiple tower networks based on the category labels of the sample text and the classification result, and to calculate the global loss of the multiple tower networks based on the mutual exclusivity between the category labels of the multiple vertical categories; and a first adjustment module, used to adjust the parameters of the tower networks based on the losses of each of the multiple tower networks and the global loss of the multiple tower networks.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to the present disclosure.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods provided in this disclosure.

[0010] According to a seventh aspect, a computer program product is provided, comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method provided in this disclosure when executed by a processor.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is an exemplary system architecture diagram illustrating a text classification method and a deep learning model training method that can be applied according to an embodiment of the present disclosure;

[0014] Figure 2 This is a flowchart of a text classification method according to an embodiment of the present disclosure;

[0015] Figure 3 This is a schematic diagram of a text classification method according to an embodiment of the present disclosure;

[0016] Figure 4 This is a flowchart of a training method for a deep learning model according to an embodiment of the present disclosure;

[0017] Figure 5 This is a schematic diagram of the structure of a deep learning model according to an embodiment of the present disclosure;

[0018] Figure 6 This is a block diagram of a text classification apparatus according to an embodiment of the present disclosure;

[0019] Figure 7 This is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure;

[0020] Figure 8 This is a block diagram of an electronic device comprising at least one of a text classification method and a deep learning model training method according to an embodiment of the present disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] Risky texts in search scenarios are diverse. Currently, the following methods are mainly used for risky text identification.

[0023] Option 1: Train and deploy a complete model independently for each vertical category of risk text recognition task. During online inference, the user's query is processed by each independent model to obtain the final risk assessment result.

[0024] Option 2: Train an end-to-end multi-label classification model using risky texts from multiple vertical categories. During online inference, the user's query is input into the model, and the model can output the probability that the query belongs to the risky text of each vertical category.

[0025] While Option 1 is highly targeted, it consumes a huge amount of resources. For example, deploying one ERNIE (Enhanced Representation through kNowledge intEgration) for each vertical category results in a single-card processing capacity of only about 30 QPS (Queries Per Second). Multiple vertical categories require the deployment of hundreds of GPUs (Graphics Processing Units) to meet peak traffic, resulting in low GPU utilization (less than 45%) and wasted resources.

[0026] Option 2 integrates resources to some extent and reduces model size. However, with multiple labels (e.g., 10) in a single model, the number of parameters is enormous, leading to a further decrease in QPS per GPU (approximately 12). It cannot support independent parameter tuning for each vertical category, and adding a new label requires full retraining, resulting in a long iteration cycle (>7 days).

[0027] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0029] Figure 1 This is a schematic diagram of an exemplary system architecture according to an embodiment of the present disclosure, showing an application of text classification methods and deep learning model training methods. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices, including but not limited to smartphones, tablets, laptops, etc.

[0032] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests and feed the processing results back to the terminal devices.

[0033] The text classification method and deep learning model training method provided in this disclosure can generally be executed by the server 105. Accordingly, the text classification device and deep learning model training device provided in this disclosure can generally be located in the server 105.

[0034] Figure 2 This is a flowchart of a text classification method according to an embodiment of the present disclosure.

[0035] like Figure 2 As shown, the text classification method 200 includes operations S210 to S230.

[0036] In operation S210, semantic features of the input text are generated using a shared pedestal network.

[0037] For example, the input text can be text entered by the user in the search box, text converted from input speech, text recognized from input images, text extracted from comments or bullet screens, text extracted from video subtitles such as live streams, etc. In other words, the embodiments of this disclosure can be applied to various application scenarios such as search display, search recommendation, image search, and voice search.

[0038] Embodiments of this disclosure may involve using a trained deep learning model to classify input text. This deep learning model may include a shared pedestal network and multiple task tower networks (hereinafter referred to as task towers or tower networks), each tower network being used to implement a specific text classification task, namely, a risk text recognition task.

[0039] For example, in a search scenario, there may be risky texts in more than ten vertical categories, such as Vertical A, Vertical B, and Vertical C. Each risky text recognition task in a vertical category can correspond to a task tower. The task tower for Vertical A is used to identify whether the input text is risky text of Vertical A, the task tower for Vertical B is used to identify whether the input text is risky text of Vertical B, and so on.

[0040] Shared base networks can be natural language processing models pre-trained in the search domain. For example, the ERNIE-Tiny (a lightweight version of ERNIE) model can be used as a shared base network, as can other pre-trained models such as BERT-base (Bidirectional Encoder Representations from Transformers) and RoBERTa (Robustly Optimized BERT Pretraining Approach). These models all demonstrate good semantic understanding capabilities in natural language processing tasks. Shared base networks pre-trained using a large number of queries from the search domain can better understand the semantic characteristics of search queries.

[0041] For example, the input text could be a query entered by the user. The query is then fed into a shared base network, which can extract the semantic features of the query.

[0042] In operation S220, the classification results of each of the multiple text classification tasks corresponding to the multiple tower networks are determined based on the semantic features and the concatenation weight matrix of the multiple tower networks.

[0043] For example, ResFNN (Residual Feedforward Neural Network) can be used as the tower network, or recurrent neural network structures such as LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit) can be used as the tower network.

[0044] The concatenated weight matrix is ​​obtained by concatenating the weight matrices of the same processing layer from multiple tower networks. For example, the processing layer is an FC (Fully Connected Layer). Each vertical category corresponds to a tower network, and each tower network can include multiple processing layers. The FC weights of the same layer from multiple tower networks can be concatenated to obtain the concatenated weight matrix of multiple tower networks.

[0045] The embodiments of this disclosure splice the FC weights of multiple tower networks at the same layer, which can merge the calculations of multiple independent fully connected layers into a single large-scale batch matrix multiplication calculation, which can be called BatchedFC (batch fully connected layer) calculation.

[0046] For example, if there are 10 tower networks, and each tower network includes 4 layers of fully connected (FC), the weight matrices of the first layer FCs of the 10 tower networks can be concatenated to obtain the concatenated weight matrix of the first layer FCs, and the weight matrices of the second layer FCs of the 10 tower networks can be concatenated to obtain the concatenated weight matrix of the second layer FCs, and so on.

[0047] The semantic features output by the shared pedestal network can be multiplied by the concatenation weight matrix of the first FC layer to obtain the output of the first FC layer. Then, the output of the first FC layer is multiplied by the concatenation weight matrix of the second FC layer, and so on, to obtain the output of the last FC layer.

[0048] In related technologies, semantic features need to be fed into the FC layer of each tower sequentially for calculation. The embodiments of this disclosure concatenate the weight matrices of the same layer FC of multiple tower networks. In each layer, the calculation of all vertical categories can be completed simultaneously through a large-scale matrix multiplication operation, which can improve the calculation efficiency and reduce the calculation latency.

[0049] For example, the serial calculation method in related technologies requires 40 multiplication operations of 10 towers multiplied by 4 layers, resulting in a large delay. However, the embodiments of this disclosure can combine the calculation of 10 independent fully connected layers of towers into 4 large matrix multiplication operations, which greatly reduces the calculation delay.

[0050] Through batch fully connected layer computation, the output of the last fully connected layer can include the classification results of all vertical classification tasks. This output, containing classification results for all verticals, can be split to obtain the classification result for each task. For example, the classification result could be the probability that the input text belongs to the risky text category for each vertical.

[0051] In operation S230, the target classification result of the input text is determined based on the classification results of multiple text classification tasks.

[0052] For example, the probability that the input text belongs to each vertical category of risky text can be used to determine which vertical category the input text ultimately belongs to.

[0053] According to the embodiments of this disclosure, by concatenating the FC weight matrices of multiple tower networks at the same layer, the semantic features of the input text are calculated with the concatenated FC weight matrices of multiple tower networks at the same layer. This allows for the simultaneous completion of the calculation of all tower networks at the same FC layer through a single matrix multiplication operation, thereby improving computational efficiency and reducing computational latency.

[0054] Figure 3 This is a schematic diagram of a text classification method according to an embodiment of the present disclosure.

[0055] like Figure 3As shown, this embodiment can employ a deep learning model for text classification. The deep learning model may include a shared pedestal network 310 and multiple tower networks 320. The multiple tower networks 320 correspond to multiple vertical text classification tasks, respectively.

[0056] The user-input query can be in text format. First, the text can be converted into tokens that the model can understand, resulting in a token sequence. Then, the token sequence is input into the shared pedestal network 310 to obtain the semantic features of the query.

[0057] Next, the semantic features can be input into multiple tower networks 320 in parallel. Within each tower network 320, the weight matrices of the FC layers (currently connected) at the same level can be pre-concatenated to obtain a concatenated weight matrix. At each FC layer, the semantic features and the concatenated weight matrix are then used for batch fully connected layer computation.

[0058] According to embodiments of this disclosure, batch fully connected layer computation includes performing parallel computation of semantic features and multiple weight matrices in a concatenated weight matrix to obtain output features; segmenting the output features into output features for multiple tasks; and determining the classification results for each of the multiple tasks based on their respective output features.

[0059] For example, a batch of queries input to a shared pedestal network 310 yields semantic features of the form [384, 384], indicating a batch size of 384 and a vector dimension of 384 for each query. The weight matrix of the FC layer of each tower network has a dimension of [384, 384]. The weight matrices of the same FC layers from multiple (e.g., 10) tower networks can be concatenated along the batch size dimension to obtain a concatenated weight matrix of the form [10, 384, 384]. Multiplying the semantic features by the concatenated weight matrix is ​​equivalent to multiplying the semantic features by 10 matrices of the form [384, 384]. By calling a single multiplication operation, 10 matrix multiplication operations are performed in parallel, completing the calculation of a large matrix and yielding an output feature of the form [10, 384, 384]. This output feature includes the calculation results from the 10 tower networks.

[0060] For example, the classification layer of the last FC layer of each tower network will convert the last dimension feature vector (384-dimensional feature vector) into 2-dimensional output. Therefore, the output features of [10, 384, 384] can be reshaped into the output features of [10, 384, 2].

[0061] Next, the output features in the form [10, 384, 2] can be segmented to obtain the output features of each tower network. The last two dimensions of the output features of each tower network represent the probability that the input text is risky text and the probability that it is not risky text, which is the initial score of the tower network output. The initial two-dimensional score of the tower network output can be processed into a final probability, and this probability can be compared with a preset threshold to determine whether the input text is risky text.

[0062] For example, the threshold for risky text in vertical A is 0.94. If the probability that the input text output by the tower network of vertical A is a risky text of vertical A is greater than 0.94, then the input text can be determined to be a risky text of vertical A; otherwise, it does not belong to the risky text of vertical A. The determination of the classification results for other verticals B, C, and N is similar to that for vertical A.

[0063] According to embodiments of this disclosure, the weight matrix of the shared pedestal network and the weight matrices of the multiple tower networks can be stored separately. For example, since the shared pedestal network is needed for each text classification, the weight matrix of the shared pedestal network can be resident in the GPU memory, while the weight matrices of the multiple tower networks can be stored in other processing units, such as the CPU.

[0064] According to embodiments of this disclosure, in response to a service requirement instruction, at least one target weight matrix is ​​determined from the weight matrices of multiple tower networks; the at least one target weight matrix is ​​then concatenated to obtain a weight concatenation matrix. For example, the at least one target weight matrix is ​​loaded into the processor where the weight matrix of the shared base network resides.

[0065] For example, different businesses have different business requirements. Some businesses may need to identify whether the input text belongs to a certain category of risky text, while others may need to identify whether the input text belongs to another category of risky text. Therefore, the weight matrix of the tower network for the required vertical category can be loaded on demand, i.e., the target weight matrix.

[0066] For example, the target weight matrix can be loaded from the CPU to the GPU where the weight matrix of the shared pedestal network is located. Then, the target weight matrix is ​​concatenated to obtain the concatenated weight matrix. The semantic features of the input text are multiplied by the concatenated weight matrix to obtain the output result.

[0067] According to embodiments of this disclosure, by keeping the weight matrix of the shared pedestal network resident in the GPU memory and loading the weight matrix of the tower network on demand, the memory usage can be reduced and the resource utilization of the hardware can be improved.

[0068] According to embodiments of this disclosure, this disclosure also provides a method for training a deep learning model.

[0069] Figure 4 This is a flowchart of a training method for a deep learning model according to an embodiment of the present disclosure.

[0070] like Figure 4 As shown, the training method 400 of this deep learning model includes operations S410 to S440. The deep learning model includes a shared pedestal network and multiple tower networks corresponding to classification tasks for multiple vertical categories.

[0071] In operation S410, sample data is input into the shared pedestal network to obtain the semantic features of the sample data.

[0072] For example, sample data can be a query in the search domain. Sample data includes multiple sample texts with category labels for multiple vertical categories, such as sample texts with labels for more than ten vertical categories, such as vertical category A, vertical category B, vertical category C, etc.

[0073] For example, if sample text has a category label for vertical A, it means that the sample text is risky text belonging to vertical A. If sample text has a category label for vertical B, it means that the sample text is risky text belonging to vertical B. In addition, the sample data includes sample text with category labels for multiple verticals, as well as sample text with normal labels. Sample text with normal labels is normal text and is risky text that does not belong to any vertical.

[0074] A shared pedestal network can be a natural language processing model pre-trained in the search domain. For example, a shared pedestal network can be one of ERNIE-Tiny, BERT-base, or RoBERTa, all pre-trained in the search domain. A shared pedestal network pre-trained using a large number of queries from the search domain can better understand the semantic features of search queries.

[0075] For example, by inputting sample text into a shared pedestal network, the shared pedestal network can extract semantic features from the sample text.

[0076] In operation S420, semantic features are input into multiple tower networks to obtain the classification results of each tower network for the sample text.

[0077] The semantic features of the sample text can be input into each tower network, and each tower network calculates the probability that the sample text belongs to the risk text of the corresponding vertical category. For example, the tower network of vertical A calculates the probability that the sample text is a risk text of vertical A, the tower network of vertical B calculates the probability that the sample text is a risk text of vertical B, and so on. In other words, for any sample, the probability that the sample text belongs to the risk text of each vertical category calculated by each tower network will be obtained.

[0078] In operation S430, the loss of each of the multiple tower networks is calculated based on the category labels and classification results of the sample text, and the global loss of the multiple tower networks is calculated based on the mutual exclusion between the category labels of the multiple vertical categories.

[0079] The sample text has true category labels. After determining the probability that the sample text belongs to each category of risky text, the loss of the tower network can be calculated based on the relationship between the true category labels and the probabilities of risky texts in each category.

[0080] To ensure that multiple tower networks optimize independently and do not affect each other, each tower network can calculate its loss independently. For example, for each tower network, the loss can be calculated by selecting sample texts with category labels belonging to the same vertical category as that tower network, while masking sample texts with category labels from other vertical categories. In this way, when training multiple tower networks simultaneously, the gradient update of one tower network will not adversely affect another tower network, avoiding the negative transfer phenomenon in multi-task learning.

[0081] Additionally, the mutual exclusion between category labels of different verticals can be used to introduce additional loss. Mutual exclusion between category labels of different verticals means that a sample text is highly unlikely to belong to two categories simultaneously. For example, if a sample text is risky text in vertical A, then it will not be risky text in vertical B.

[0082] Based on the mutual exclusivity of category labels, the label extrapolation loss for each sample text can be calculated. For example, the label extrapolation loss can be determined based on the constraint that, given a sample text is determined to be a risky text in vertical A, the probability of that sample text belonging to other risky texts in vertical A should be as small as possible. The label extrapolation loss is calculated for sample texts in each vertical category and is a global loss belonging to multiple tower networks.

[0083] When operating the S440, adjust the parameters of the tower network based on the losses.

[0084] For example, the loss calculated independently for each tower network can be used only to update the loss of that specific tower network. However, the label extrapolation loss calculated based on the mutual exclusion of category labels is the global loss and can be used to update the loss of all tower networks.

[0085] According to embodiments of this disclosure, sample text is input into a shared pedestal network to obtain semantic features. The semantic features are then input into all tower networks to obtain the classification result of each tower network for the sample text. The loss of each tower network is calculated based on the classification result of the sample text and the category label. The global loss is calculated based on the mutual exclusion between the category labels. The tower networks are updated based on the loss of each tower network and the global loss. This allows each tower network to be optimized independently and improves the overall performance of multiple tower networks.

[0086] The following is combined Figure 5 The training method of the deep learning model according to the embodiments of this disclosure will be further described.

[0087] Figure 5 This is a schematic diagram of the structure of a deep learning model according to an embodiment of the present disclosure.

[0088] like Figure 5 As shown, the deep learning model in this embodiment includes a shared pedestal network 510 and multiple tower networks 521-523. Each tower network may include multiple fully connected layers (FC), with a classification function connected to the last FC layer, outputting the probability of whether a sample text belongs to the corresponding vertical category of risky text.

[0089] The training of the deep learning model in this embodiment can include two stages. The first stage is unsupervised domain-adaptive training, and the second stage is multi-task fine-tuning.

[0090] According to embodiments of this disclosure, the shared base network is pre-trained using sample text from a specific domain, which includes at least one of search display, search recommendation, image search, and voice search.

[0091] In the first stage, a large amount of sample text from the search domain can be used to perform post-pretraining on the pre-trained natural language processing model to obtain a shared base network suitable for the search domain. The natural language processing model can be one of ERNIE-Tiny, BERT-base, or RoBERTa.

[0092] For example, query text can be collected from query logs in various scenarios such as search display, search recommendation, image search, and voice search. Then, the collected query text is cleaned, such as removing duplicate queries, filtering queries with a length less than a threshold (e.g., 4 characters), removing queries containing only data or URLs, removing queries with low exposure (e.g., less than 5 exposures), and removing queries containing control characters, to obtain cleaned queries, which are used as sample text.

[0093] Next, the sample text can be segmented into words. From the resulting large number of segments, 10% are selected for whole word masking (WWM) training. The trained shared pedestal network can better understand the semantic features of the search query.

[0094] In the second stage, multiple category labels can be assigned to the sample text, such as category A, category B, category C, and more than ten other categories. Each category yields sample text with that category label. In addition, the sample text also includes normal text with standard labels.

[0095] After preparing sample texts for each vertical category, a batch of sample texts can be input into the deep learning model. For each sample text, it can first pass through a shared pedestal network to obtain semantic features. Then, the semantic features are input into the tower networks of each vertical category to obtain the probability that each tower network classifies the sample text as a risky text belonging to the corresponding vertical category. In other words, during the forward propagation process, each tower network calculates the classification result for all sample texts.

[0096] However, when calculating the loss, each tower network calculates the loss independently and backpropagates independently to update its own parameters.

[0097] According to embodiments of this disclosure, for each tower network, the loss of the tower network is determined based on the classification result of the tower network for the target sample text, wherein the target sample text has a category label of the vertical class corresponding to the tower network.

[0098] For example, a Mask Label Loss strategy can be used to calculate the independent loss for each tower network. Specifically, for each tower network, if a sample does not have a category label of the vertical to which that tower network belongs, then that sample is not included in the loss calculation for that tower network. In other words, each tower network only uses sample text with category labels belonging to the same vertical as that tower network to calculate the loss, while masking sample text with category labels of other verticals.

[0099] For example, when calculating the loss of a pyramid network for vertical category A, sample text with category labels of vertical category A is used as the target sample text. The loss is calculated based on the difference between the classification result of the pyramid network for vertical category A and the category label of the target sample text. The sum of the losses of all sample texts with category labels of vertical category A can be determined as the loss of the pyramid network for vertical category A. The loss calculation method for pyramid networks of other vertical categories is similar.

[0100] Each tower network adopts the Mask Label Loss strategy to obtain its own loss. The loss of each tower network is only used to update the parameters of the tower network itself, which enables each tower network to optimize and iterate independently, avoiding the negative transfer phenomenon in multi-task learning caused by mutual influence between tower networks.

[0101] According to embodiments of this disclosure, for each vertical category of sample text, the label extrapolation loss of the sample text is determined based on the classification results of the sample text by other tower networks besides the tower network of the vertical category; and the global loss of multiple tower networks is determined based on the label extrapolation loss of the sample text.

[0102] The embodiments of this disclosure also employ a label extrapolation strategy to construct additional losses. The label extrapolation strategy means that, given a sample text is determined to be risky text of a specific vertical category, the probability that the sample text belongs to risky text of other vertical categories should be as small as possible.

[0103] For each vertical category, the sample text belonging to that vertical category has a category label for that vertical category. The label extrapolation loss can be calculated for the sample text of each vertical category. For example, the label extrapolation loss for vertical category i can be calculated using the following formula (1).

[0104] (1)

[0105] Where i represents the i-th vertical category, j represents the j-th vertical category, and i and j are both integers greater than or equal to 1 and less than or equal to N, where N is the total number of vertical categories. Lextra represents the label extrapolation loss, and y_i represents the true label of the sample. If the sample belongs to vertical i, then y_i = 1; otherwise, y_i is 0. p_j represents the initial score of the risk text of vertical j output by the model. σ is the classification function used to map the initial score output by the model to the range of 0-1.

[0106] The goal of label extrapolation is to use the prior knowledge of "mutual exclusion of categories" to enable the model to learn that "if a sample belongs to category i, it should try not to belong to other categories j".

[0107] For example, when the true label of a sample is i, y_i=1, the predicted probability for all other non-i categories needs to be as small as possible (i.e., p_j should be small). In this case, if the predicted probability for category j is large, p_j is larger, σ(p_j) approaches 1, 1-σ(p_j) approaches 0, and -log(1-σ(p_j)) approaches positive infinity. Therefore, this loss function can effectively penalize the situation where a sample of category i is incorrectly predicted as a sample of another category.

[0108] The label extrapolation loss of all verticals can be calculated based on the above formula (1), and then the sum of the label extrapolation losses of all verticals can be used as the global loss of all tower networks.

[0109] According to embodiments of this disclosure, for each tower network, the parameters of the tower network itself are adjusted based on the tower network's own losses; the parameters of multiple tower networks are adjusted based on global losses.

[0110] The loss for each tower network is calculated independently and used only to update the loss of that tower network itself. The label extrapolation loss is the global loss across multiple tower networks; therefore, the label extrapolation loss can be used to update the loss of all tower networks.

[0111] In one example, when adjusting the tower network parameters, you can adjust only the parameters of the classification layer of the tower network.

[0112] According to embodiments of this disclosure, the parameters of a shared base network are adjusted with a preset probability based on the individual losses of the multiple tower networks and the global losses of the multiple tower networks.

[0113] In the second phase of multi-task fine-tuning, the shared base station network can also be fine-tuned based on the tower network loss and the label extrapolation loss. For example, the loss of the shared base station network can be updated using the sum of the losses of multiple tower networks and the label extrapolation loss as the overall loss.

[0114] Since the multi-task fine-tuning phase is primarily used to train multiple tower networks, significant changes to the shared pedestal network are undesirable. Therefore, a pedestal freeze rate of 50% can be set, meaning the parameters of the shared pedestal network are adjusted with a 50% probability based on the overall loss mentioned above.

[0115] In the embodiments of this disclosure, each tower network independently calculates its own loss, adjusts its own parameters based on its own loss, and calculates the global label extrapolation loss of all tower networks based on class mutual exclusion. The parameters of all tower networks are adjusted using the label extrapolation loss. In addition, the parameters of the shared base are adjusted with a preset probability based on the individual losses of multiple tower networks and the global label extrapolation loss. This ensures that each tower network is optimized independently and improves the overall performance of the deep learning model.

[0116] According to embodiments of this disclosure, this disclosure also provides a text classification device and a training device for a deep learning model.

[0117] Figure 6 This is a block diagram of a text classification apparatus according to an embodiment of the present disclosure.

[0118] like Figure 6 As shown, the text classification device 600 includes a first semantic feature determination module 610, a first classification module 620, and a classification result determination module 630.

[0119] The first semantic feature determination module 610 is used to generate semantic features of the input text using a shared base network.

[0120] The first classification module 620 is used to determine the classification results of multiple text classification tasks corresponding to multiple tower networks based on the semantic features and the concatenated weight matrix of multiple tower networks. The concatenated weight matrix is ​​obtained by concatenating the weight matrices of the same processing layer of multiple tower networks.

[0121] The classification result determination module 630 is used to determine the target classification result of the input text based on the classification results of multiple text classification tasks.

[0122] The first classification module 620 includes a parallel computing unit, a segmentation unit, and a classification result determination unit.

[0123] The parallel computing unit is used to perform parallel computation on semantic features and multiple weight matrices in the concatenated weight matrix to obtain output features.

[0124] The segmentation unit is used to segment the output features into the output features of each of the multiple tasks.

[0125] The classification result determination unit is used to determine the classification result of each of the multiple tasks based on their respective output features.

[0126] According to embodiments of this disclosure, the text classification device 600 further includes a target weight matrix determination module and a splicing module.

[0127] The target weight matrix determination module is used to determine at least one target weight matrix from the weight matrices of multiple tower networks in response to business requirement indications.

[0128] The splicing module is used to splice at least one target weight matrix to obtain a weight splicing matrix.

[0129] The weight matrix of the shared base network is stored separately from the weight matrix of the tower network. The target weight matrix determination module is used to load at least one target weight matrix into the processor where the weight matrix of the shared base network resides, in response to a business requirement instruction.

[0130] Figure 7 This is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure.

[0131] like Figure 7 As shown, the training device 700 of the deep learning model includes a second semantic feature determination module 710, a second classification module 720, a loss determination module 730, and a first adjustment module 740.

[0132] The second semantic feature determination module 710 is used to input sample data into the shared base network to obtain the semantic features of the sample data, wherein the sample data includes multiple sample texts with category labels of multiple vertical categories.

[0133] The second classification module 720 is used to input semantic features into multiple tower networks to obtain the classification results of each tower network for the sample text.

[0134] The loss determination module 730 is used to calculate the loss of each of the multiple tower networks based on the category labels and classification results of the sample text, and to calculate the global loss of the multiple tower networks based on the mutual exclusion between the category labels of the multiple verticals.

[0135] The first adjustment module 740 is used to adjust the parameters of the tower networks based on the individual losses of the multiple tower networks and the global loss of the multiple tower networks.

[0136] The loss determination module 730 is also used to determine the loss of each tower network based on the classification result of the tower network for the target sample text, wherein the target sample text has a category label of the vertical class corresponding to the tower network.

[0137] The loss determination module 730 includes a label extrapolation loss determination unit and a global loss determination unit.

[0138] The label extrapolation loss determination unit is used to determine the label extrapolation loss of the sample text for each vertical class based on the classification results of the sample text by other tower networks besides the tower network of the vertical class.

[0139] The global loss determination unit is used to extrapolate the loss based on the labels of the sample text to determine the global loss of multiple tower networks.

[0140] The first adjustment module 740 includes a first adjustment unit and a second adjustment unit.

[0141] The first adjustment unit is used to adjust the parameters of each tower network based on its own losses.

[0142] The second adjustment unit is used to adjust the parameters of multiple tower networks based on the global loss.

[0143] The training device 700 for the deep learning model also includes a second adjustment module.

[0144] The second adjustment module is used to adjust the parameters of the shared base network with a preset probability based on the individual losses of the multiple tower networks and the global losses of the multiple tower networks.

[0145] According to embodiments of this disclosure, the text is obtained through pre-training using sample text from a specific domain, which includes at least one of search display, search recommendation, image search, and voice search.

[0146] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0147] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0148] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0149] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0150] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs at least one of the various methods and processes described above, such as text classification methods and deep learning model training methods. For example, in some embodiments, at least one of the text classification methods and deep learning model training methods can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of at least one of the text classification methods and deep learning model training methods described above can be performed. Alternatively, in other embodiments, computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform at least one of the text classification method and the training method of the deep learning model.

[0151] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0155] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0156] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0157] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A text classification method, comprising: Semantic features of input text are generated using a shared pedestal network; Based on the semantic features and the concatenated weight matrix of multiple tower networks, the classification results of multiple text classification tasks corresponding to the multiple tower networks are determined respectively. The concatenated weight matrix is ​​obtained by concatenating the weight matrices of the same processing layer of the multiple tower networks. as well as The target classification result of the input text is determined based on the classification results of the multiple text classification tasks.

2. The method according to claim 1, wherein, The step of determining the classification results of each of the multiple text classification tasks corresponding to the multiple tower networks based on the concatenated weight matrix of the semantic features and the multiple tower networks includes: The semantic features are computed in parallel with multiple weight matrices in the concatenated weight matrix to obtain the output features; The output features are segmented into the output features of the multiple tasks respectively; and Based on the output characteristics of each of the multiple tasks, the classification results of each of the multiple tasks are determined.

3. The method according to claim 1, further comprising: In response to a business requirement instruction, at least one target weight matrix is ​​determined from the weight matrices of the plurality of tower networks; as well as The at least one target weight matrix is ​​concatenated to obtain the weight concatenation matrix.

4. The method according to claim 1, wherein, The weight matrix of the shared base network is stored separately from the weight matrix of the tower network; The step of determining at least one target weight matrix from the weight matrices of the plurality of tower networks in response to a service demand indication includes: In response to a business requirement instruction, the at least one target weight matrix is ​​loaded into the processor where the weight matrix of the shared base network resides.

5. A method for training a deep learning model, wherein, The deep learning model includes a shared pedestal network and multiple tower networks corresponding to classification tasks for multiple vertical categories; the method includes: The sample data is input into the shared base network to obtain the semantic features of the sample data, wherein the sample data includes multiple sample texts with category labels of the multiple vertical categories; The semantic features are input into the multiple tower networks to obtain the classification result of each tower network for the sample text; The loss of each of the multiple tower networks is calculated based on the category labels of the sample text and the classification results; and the global loss of the multiple tower networks is calculated based on the mutual exclusion between the category labels of the multiple vertical categories; and The parameters of the tower networks are adjusted based on the individual losses of the multiple tower networks and the global loss of the multiple tower networks.

6. The method according to claim 5, wherein, The calculation of the loss of each of the multiple tower networks based on the category label of the sample text and the classification result includes: For each tower network, the loss of the tower network is determined based on the classification result of the tower network for the target sample text, wherein the target sample text has a category label of the vertical class corresponding to the tower network.

7. The method according to claim 5, wherein, The step of calculating the global loss of the multiple tower networks based on the mutual exclusion among the category labels of the multiple vertical categories includes: For each vertical category of sample text, the label extrapolation loss of the sample text is determined based on the classification results of the sample text by other pyramid networks besides the pyramid network for that vertical category; and The global loss of the multiple tower networks is determined based on the label extrapolation loss of the sample text.

8. The method according to claim 5, wherein, The step of adjusting the parameters of the tower networks based on the individual losses of the multiple tower networks and the global loss of the multiple tower networks includes: For each tower network, adjust the parameters of the tower network itself based on its own losses; and The parameters of the multiple tower networks are adjusted based on the global loss.

9. The method according to claim 5, further comprising: The parameters of the shared base network are adjusted with a preset probability based on the individual losses of the multiple tower networks and the global losses of the multiple tower networks.

10. The method according to claim 5, wherein, The shared pedestal network is pre-trained using sample text from a specific domain, which includes at least one of search display, search recommendation, image search, and voice search.

11. A text classification device, comprising: The first semantic feature determination module is used to generate semantic features of the input text using a shared base network; The first classification module is used to determine the classification results of multiple text classification tasks corresponding to the multiple tower networks based on the semantic features and the concatenated weight matrix of the multiple tower networks. The concatenated weight matrix is ​​obtained by concatenating the weight matrices of the same processing layer of the multiple tower networks. The classification result determination module is used to determine the target classification result of the input text based on the classification results of the multiple text classification tasks.

12. A training device for a deep learning model, wherein, The deep learning model includes a shared pedestal network and multiple tower networks corresponding to classification tasks for multiple vertical categories; the device includes: The second semantic feature determination module is used to input the sample text into the shared base network to obtain the semantic features of the sample text, wherein the sample text includes multiple sample texts with category labels of the multiple vertical categories; The second classification module is used to input the semantic features into the multiple tower networks to obtain the classification result of each tower network for the sample text; A loss determination module is used to calculate the loss of each of the multiple tower networks based on the category labels of the sample text and the classification results, and to calculate the global loss of the multiple tower networks based on the mutual exclusion between the category labels of the multiple vertical categories; and The first adjustment module is used to adjust the parameters of the tower networks based on the individual losses of the multiple tower networks and the global loss of the multiple tower networks.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method according to any one of claims 1 to 10 when executed by a processor.

Images