Text classification method and device, computer device and storage medium

Abstract

This application relates to a text classification method, apparatus, computer device, and storage medium. The method includes: inputting sample texts into a text multi-classification model to be trained to obtain text feature vectors; for each text binary classifier in the text multi-classification model, treating sample texts of the target class as positive samples and sample texts of non-target classes as negative samples, and normalizing the classifier weight vector and the text feature vectors of the positive and negative samples to a unit hypersphere; determining an angular feature loss function; substituting the angle between the normalized text feature vectors of the positive and negative samples and the classifier weight vector into the angular feature loss function to obtain a sub-loss value corresponding to the text binary classifier; determining a target loss value based on the sub-loss values ​​corresponding to each text binary classifier to be trained, and iteratively optimizing the training in the direction of minimizing the target loss value to obtain the trained text multi-classification model. This method can improve the classification accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a text classification method and device, computer equipment and a storage medium. BACKGROUND

[0002] As a main method for solving the intent recognition task, text classification is one of the most important and basic tasks in natural language processing. Through computer automatic learning of text features, the classification process is automated, and the waste of unnecessary human resources in manual classification is reduced.

[0003] With the development of deep learning technology, more and more deep models are applied to the field of text classification. The traditional deep learning method is to determine the loss based on the difference between the prediction result and the label through linear calculation, so as to optimize and train the classification model. This way will cause the classification discrimination performance of the trained text classification model to be relatively low when training the text classification model of the multi-classification task, affecting the accuracy of text classification. SUMMARY

[0004] Therefore, it is necessary to provide a text classification method, device, computer equipment, computer readable storage medium and computer program product capable of improving the accuracy of text classification in view of the above technical problems.

[0005] In a first aspect, the present application provides a text classification method. The method comprises:

[0006] obtaining a sample text set; the sample text set comprises sample texts of a plurality of preset categories;

[0007] inputting each sample text into a text multi-classification model to be trained to extract a text feature vector of each sample text; the text multi-classification model to be trained comprises a plurality of text binary classifiers to be trained; each text binary classifier corresponds to a target category of the plurality of preset categories;

[0008] for each text binary classifier, sample texts belonging to the corresponding target category are regarded as positive samples, and sample texts not belonging to the target category are regarded as negative samples, and the classifier weight vector of the text binary classifier and the text feature vector corresponding to the positive and negative samples are normalized to a unit hypersphere;

[0009] determining an angle feature loss function; the angle feature loss function comprises a positive and negative sample balance parameter;

[0010] substitute the angle between the normalized text feature vector of the positive and negative samples and the classifier weight vector into the angle feature loss function respectively to obtain a sub-loss value corresponding to the text binary classifier;

[0011] determine a target loss value based on the sub-loss value corresponding to each text binary classifier to be trained, and iteratively optimize training in a direction of minimizing the target loss value to obtain a trained text multi-classification model.

[0012] In one of the embodiments, the angle feature loss function further comprises a similarity calculation function constructed based on an angle variable; the similarity calculation function is used to calculate a cosine similarity between the text feature vectors normalized to a unit hypersphere and the classifier weight vectors; and the angle variable is used to represent an angle between the text feature vectors and the classifier weight vectors.

[0013] The angle between each of the normalized text feature vectors of the positive and negative samples and the classifier weight vectors is substituted into the angle feature loss function to obtain the sub-loss value corresponding to the text binary classifier, which comprises:

[0014] The angle between the text feature vectors normalized to the unit hypersphere and the classifier weight vectors is substituted into the similarity calculation function to calculate the cosine similarity corresponding to each of the positive and negative samples respectively.

[0015] The loss value corresponding to each of the positive and negative samples respectively is determined according to the cosine similarity corresponding to each of the positive and negative samples respectively.

[0016] The sub-loss value corresponding to the text binary classifier is obtained according to the loss value corresponding to each of the positive and negative samples respectively and the positive and negative sample balance parameter.

[0017] In one of the embodiments, the angle feature loss function further comprises a similarity mapping function; and the similarity calculation function is a sub-function of the similarity mapping function.

[0018] The loss value corresponding to each of the positive and negative samples respectively is determined according to the cosine similarity corresponding to each of the positive and negative samples respectively, which comprises:

[0019] The cosine similarity calculated is mapped by the similarity mapping function to obtain a mapping result corresponding to each of the positive and negative samples respectively.

[0020] The loss value corresponding to each of the positive and negative samples respectively is determined based on the mapping result corresponding to each of the positive and negative samples respectively.

[0021] In one of the embodiments, the similarity mapping function further comprises a similarity adjustment parameter; and the similarity adjustment parameter is used to constrain the overlapping degree between the cosine similarity distributions corresponding to each of the positive and negative samples respectively.

[0022] In one of the embodiments, the angle feature loss function comprises a first angle margin parameter and a second angle margin hyper-parameter; the first angle margin parameter is used to control the angle margin size of the positive sample; and the second angle margin hyper-parameter is used to control the angle margin size of the negative sample.

[0023] The loss value corresponding to each of the positive sample and the negative sample is determined according to the cosine similarity corresponding to each of the positive sample and the negative sample.

[0024] For the positive sample, the loss value corresponding to the positive sample is determined according to the cosine similarity corresponding to the positive sample and the first angle margin parameter.

[0025] For the negative sample, the loss value corresponding to the negative sample is determined according to the cosine similarity corresponding to the negative sample and the second angle margin hyper-parameter.

[0026] In one of the embodiments, the angle between the normalized text feature vector of each of the positive sample and the negative sample and the classifier weight vector is substituted into the angle feature loss function to obtain the sub-loss value corresponding to the text binary classifier.

[0027] The angle between the normalized text feature vector of each of the positive sample and the negative sample and the classifier weight vector is substituted into the angle feature loss function to obtain the loss value corresponding to each of the positive sample and the negative sample.

[0028] The loss value corresponding to each of the positive sample and the negative sample is weighted according to the weight corresponding to each of the positive sample and the negative sample to obtain the sub-loss value corresponding to the text binary classifier.

[0029] The weight corresponding to each of the positive sample and the negative sample is determined according to the positive-negative sample balance parameter in the angle feature loss function; and the weight corresponding to the positive sample is greater than the weight corresponding to the negative sample.

[0030] In one of the embodiments, the angle feature loss function further comprises a difficult sample mining parameter.

[0031] The method further comprises:

[0032] The weight corresponding to each of the positive sample and the negative sample is determined according to the positive-negative sample balance parameter and the difficult sample mining parameter in the angle feature loss function.

[0033] In a second aspect, the present application further provides a text classification device. The device comprises:

[0034] The feature extraction module is configured to obtain a sample text set, wherein the sample text set comprises sample texts of a plurality of preset categories; input each sample text into a text multi-classification model to be trained to extract a text feature vector of each sample text; and the text multi-classification model to be trained comprises a plurality of text binary classifiers to be trained, and each text binary classifier corresponds to a target category of the plurality of preset categories.

[0035] The normalization module is configured to, for each text binary classifier, normalize a classifier weight vector of the text binary classifier and text feature vectors corresponding to positive and negative samples to a unit hypersphere, wherein the positive samples belong to the corresponding target category and the negative samples do not belong to the target category.

[0036] The optimization module is configured to determine an angle feature loss function, wherein the angle feature loss function comprises a positive and negative sample balance parameter; input angles between the normalized text feature vectors of the positive and negative samples and the classifier weight vector into the angle feature loss function to obtain a sub-loss value corresponding to the text binary classifier; determine a target loss value based on the sub-loss values corresponding to the text binary classifiers to be trained, and iteratively optimize training in a direction of minimizing the target loss value to obtain a trained text multi-classification model.

[0037] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0038] obtain a sample text set, wherein the sample text set comprises sample texts of a plurality of preset categories;

[0039] input each sample text into a text multi-classification model to be trained to extract a text feature vector of each sample text, wherein the text multi-classification model to be trained comprises a plurality of text binary classifiers to be trained, and each text binary classifier corresponds to a target category of the plurality of preset categories;

[0040] for each text binary classifier, normalize a classifier weight vector of the text binary classifier and text feature vectors corresponding to positive and negative samples to a unit hypersphere, wherein the positive samples belong to the corresponding target category and the negative samples do not belong to the target category.

[0041] determine an angle feature loss function, wherein the angle feature loss function comprises a positive and negative sample balance parameter;

[0042] the angle feature loss function, to obtain a sub-loss value corresponding to the text binary classifier;

[0043] determine a target loss value based on the sub-loss value corresponding to each text binary classifier to be trained, and iteratively optimize training in a direction of minimizing the target loss value, to obtain the trained text multi-classification model.

[0044] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the following steps:

[0045] obtain a sample text set; the sample text set includes sample texts of a plurality of preset categories;

[0046] input each sample text into a text multi-classification model to be trained to extract a text feature vector of each sample text; the text multi-classification model to be trained includes a plurality of text binary classifiers to be trained; each text binary classifier corresponds to a target category of the plurality of preset categories;

[0047] for each text binary classifier, sample texts belonging to the corresponding target category are regarded as positive samples, and sample texts not belonging to the target category are regarded as negative samples, and a classifier weight vector of the text binary classifier and a text feature vector corresponding to each of the positive and negative samples are normalized to a unit hypersphere;

[0048] determine an angle feature loss function; the angle feature loss function includes a positive and negative sample balance parameter;

[0049] the angle feature loss function, to obtain a sub-loss value corresponding to the text binary classifier;

[0050] determine a target loss value based on the sub-loss value corresponding to each text binary classifier to be trained, and iteratively optimize training in a direction of minimizing the target loss value, to obtain the trained text multi-classification model.

[0051] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and the computer program, when executed by a processor, implements the following steps:

[0052] obtain a sample text set; the sample text set includes sample texts of a plurality of preset categories;

[0053] Each of the sample texts is input into a text multi-classification model to be trained to extract the text feature vector of each sample text; the text multi-classification model to be trained includes multiple text binary classifiers to be trained; each text binary classifier corresponds to one of the target categories of the multiple preset categories;

[0054] For each of the text binary classifiers, sample texts belonging to the corresponding target category are taken as positive samples, and sample texts not belonging to the target category are taken as negative samples. The classifier weight vector of the text binary classifier and the text feature vectors corresponding to the positive and negative samples are normalized to the unit hypersphere.

[0055] Determine the angle feature loss function; the angle feature loss function includes a positive and negative sample balance parameter;

[0056] Substitute the angles between the normalized text feature vectors of the positive and negative samples and the classifier weight vectors into the angle feature loss function to obtain the sub-loss values ​​corresponding to the text binary classifier.

[0057] The target loss value is determined based on the sub-loss value corresponding to each text binary classifier to be trained, and the training is iteratively optimized in the direction of minimizing the target loss value to obtain the trained text multi-classification model.

[0058] The aforementioned text classification method, apparatus, computer equipment, storage medium, and computer program product input each sample text into a text multi-classification model to be trained to extract the text feature vectors of each sample text; for each text binary classifier in the text multi-classification model to be trained, sample texts belonging to the corresponding target category are taken as positive samples, and sample texts not belonging to the target category are taken as negative samples, and the classifier weight vector of the text binary classifier and the text feature vectors corresponding to the positive and negative samples are normalized to a unit hypersphere; an angular feature loss function is determined; the angular feature loss function includes a positive and negative sample balance parameter; the angle between each of the normalized text feature vectors of the positive and negative samples and the classifier weight vector is substituted into the angular feature loss function to obtain the sub-loss value corresponding to the text binary classifier; based on the sub-loss value corresponding to each text binary classifier to be trained, a target loss value is determined, and the training is iteratively optimized in the direction of minimizing the target loss value to obtain the trained text multi-classification model. By transforming a multi-class classification task into multiple binary classification tasks and normalizing the text feature vectors and classifier weights onto a hypersphere, and then training the classifier on a unit hypersphere using an angle feature loss function, the classifier can learn more discriminative angle features on the hypersphere, thereby enhancing the text binary classifier's ability to distinguish text. Furthermore, by optimizing the loss function based on a positive-negative sample balancing parameter, the training process pays more attention to the minority of positive samples, achieving a positive-negative sample balancing adjustment and further improving text classification accuracy. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating a text classification method in one embodiment;

[0060] Figure 2 This is a schematic diagram illustrating the compression of the positive sample space and the relaxation of the negative sample space restrictions in one embodiment;

[0061] Figure 3 This is a simplified diagram illustrating the principle of a text classification method in one embodiment;

[0062] Figure 4 This is a structural block diagram of a text classification device in one embodiment;

[0063] Figure 5 This is an internal structural diagram of a computer device in one embodiment;

[0064] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] In one embodiment, such as Figure 1 As shown, a text classification method is provided. Taking the application of this method to a computer device as an example, the computer device can be a terminal or a server. The method can be implemented by the terminal or server itself, or it can be implemented through the interaction between the terminal and the server. Specifically, it includes the following steps:

[0067] Step 102: Obtain the sample text set; the sample text set includes sample texts of multiple preset categories.

[0068] It is understandable that, since this application aims to train a text multi-classification model capable of performing multi-classification tasks, the sample text set used to train the text multi-classification model includes sample texts of multiple preset categories.

[0069] In some embodiments, the sample texts in the sample text set may be telecommunications work order texts, each with a corresponding intent category label. The classification task performed by the text classification model is to identify the intent category in the text.

[0070] For ease of understanding, Table 1 lists the text of the telecommunications work order for illustrative purposes.

[0071] Table 1

[0072] Telecommunications conversation order text Intention category label Inquire about the monthly deduction details of mobile phone charges Balance and arrears doubts See how much the monthly consumption is, now 89 yuan in arrears Balance and arrears doubts What does the 99 yuan additional package fee refer to Value-added cost doubts Ask about the mobile phone number Inquire about the incoming call number Help check the 199 yuan unlimited package monthly payment of 270 yuan or more Monthly package fee doubts

[0073] In some embodiments, the text multi-classification model trained by the method of this application is used in the context of multi-turn dialogue in telecommunications customer service to intelligently determine the user's intent category label. That is, the text multi-classification model is used to predict and identify the user's intent category based on the dialogue text with the user.

[0074] Step 104: Input each sample text into the text multi-classification model to be trained to extract the text feature vector of each sample text; the text multi-classification model to be trained includes multiple text binary classifiers to be trained; each text binary classifier corresponds to one of the target categories of multiple preset categories.

[0075] Among them, the text binary classifier is a text classifier used to perform binary classification tasks. The text multi-classification model to be trained includes multiple text binary classifiers to be trained. Each text binary classifier has a corresponding target category, which is one of the multiple preset categories. The target category corresponding to the text binary classifier refers to the preset category that the text binary classifier is used to identify.

[0076] It can be understood that this application is equivalent to splitting the multi-classification task into multiple binary classification tasks. By the method of this application, multiple text binary classifiers are trained to complete the training of the entire text multi-classification model. For example, if the text multi-classification model is used to identify K preset categories, then it is split into K binary classification tasks, and there are K text binary classifiers to be trained. Each text binary classifier is used to identify one of the preset categories.

[0077] Exemplarily, the text multi-classification model has a feature extraction layer. After inputting each sample text into the text multi-classification model to be trained, the text feature vector of each sample text can be extracted through the feature extraction layer.

[0078] Exemplarily, the feature extraction layer includes a bert-base encoding model (i.e., a bidirectional encoder based on the transformer architecture), and the preprocessed bert-base encoding model can be encoded to obtain a vector x from the last hidden layer of the model. i As the word vector corresponding to each character, its expression is as follows:

[0079] x i = BERT(CLS, t l , t2... t N , SEP)

[0080] It can be understood that the word vectors of each character in the sample text constitute the text feature vector of the sample text.

[0081] In some embodiments, the computer device can perform data preprocessing on the sample texts in the sample text set and input the preprocessed texts into the text multi-classification model to be trained. Exemplarily, the computer device tokenizes the sample text by character, adds a character "[CLS]" indicating the start of the text at the beginning of the sample text, and adds a special character "[SEP]" indicating the end of the text at the end of the sample text to complete the data preprocessing and obtain the sample text. Taking the sample text "Query the detailed monthly deduction of mobile phone bills" as an example, the preprocessed text is ["[CLS]", "查", "询", "一", "下", "手", "机", "话", "费", "月", "扣", "详", "细", "[SEP]"].

[0082] Step 106: For each text binary classifier, take the sample text belonging to the corresponding target category as positive samples and the sample text not belonging to the target category as negative samples, and normalize the classifier weight vector of the text binary classifier and the text feature vectors corresponding to the positive and negative samples to the unit hypersphere.

[0083] It should be noted that since each text binary classifier corresponds to a different target category, i.e., the preset category to be identified is different, the division of positive and negative samples differs between different text binary classifiers. For each text binary classifier, the target category corresponding to that text binary classifier is determined. Sample texts belonging to that target category in the sample text set are treated as positive samples (i.e., positive sample labels are added), and sample texts belonging to other categories (i.e., preset categories other than the target category) are treated as negative samples (i.e., negative sample labels are added).

[0084] The computer device can normalize the classifier weight vector and the corresponding text feature vectors of the positive and negative samples of the text binary classifier to a unit hypersphere, so that subsequent classification and discrimination processing can be performed based on the unit hypersphere. That is, the subsequent classification and discrimination optimization process is constrained to the unit hypersphere. In the text binary classifier, the classifier parameters W... i T and text vector features x i Normalization yields |W i T |=|x i |=1.

[0085] The unit hypersphere is a geometrically uniformly distributed manifold. Normalizing the classifier weight vector and the text feature vectors of positive and negative samples to the unit hypersphere makes the distances between different samples more balanced, which can better capture the intrinsic structure of the data. Classification training based on the unit hypersphere can enhance the discriminative ability of text binary classifiers.

[0086] Step 108: Determine the angle feature loss function; the angle feature loss function includes a positive and negative sample balance parameter.

[0087] It can be understood that by normalizing the classifier weight vector and text feature vector to a unit hypersphere, the classifier's classification and discrimination optimization process is constrained to the unit hypersphere. This allows the model's optimization training to be solely related to angular features, which include the angles of the classifier weight vector and text feature vectors in the hypersphere's feature space. Therefore, an angular feature loss function can be constructed. This angular feature loss function uses angular features to reflect the difference between the classification prediction result and the label, enabling iterative optimization training of the text binary classifier using these angular features.

[0088] The positive / negative sample balancing parameter included in the angular feature loss function is used to adjust the attention given to an imbalanced number of positive and negative samples, thereby achieving a balance in the attention given to positive and negative samples. It can be understood that for a text binary classifier, only the sample text belonging to the target class corresponding to that text binary classifier is a positive sample; all other classes are negative samples. That is, out of K preset classes of sample text, only one preset class of sample text is a positive sample, and the remaining K-1 preset class of sample text are all negative samples. This leads to a large number of negative samples, thus affecting the training results. Therefore, by adding the positive / negative sample balancing parameter λ, more weight is assigned to the minority of positive samples, and less weight is assigned to the majority of negative samples, thereby reducing the attention given to negative samples and achieving a balance in the attention given to positive and negative samples.

[0089] Step 110: Substitute the angles between the normalized text feature vectors of the positive and negative samples and the classifier weight vectors into the angle feature loss function to obtain the sub-loss values ​​corresponding to the text binary classifier.

[0090] In some embodiments, step 110 includes: substituting the angle between the normalized text feature vector and the classifier weight vector corresponding to the positive and negative samples into the angle feature loss function to obtain the loss values ​​corresponding to the positive and negative samples respectively; and weighting the loss values ​​corresponding to the positive and negative samples according to the weights corresponding to the positive and negative samples respectively to obtain the sub-loss values ​​corresponding to the text binary classifier.

[0091] The weights corresponding to the positive and negative samples are determined based on the positive and negative sample balance parameter in the angle feature loss function; the weight of the positive sample is greater than the weight of the negative sample.

[0092] For example, the angle feature loss function includes a first loss function for calculating the loss value of positive samples and a second loss function for calculating the loss value of negative samples. When calculating the sub-loss value corresponding to each text binary classifier, the computer device can determine the angle between the normalized text feature vector corresponding to the positive sample and the classifier weight vector of the text binary classifier, thus obtaining the angle corresponding to the positive sample; and determine the angle between the normalized text feature vector corresponding to the negative sample and the classifier weight vector of the text binary classifier, thus obtaining the angle corresponding to the negative sample. The computer device can substitute the angle corresponding to the positive sample into the first loss function in the angle feature loss function to obtain the loss value corresponding to the positive sample, and substitute the angle corresponding to the negative sample into the second loss function in the angle feature loss function to obtain the loss value corresponding to the negative sample. Furthermore, the computer device can weight the loss values ​​corresponding to the positive and negative samples according to the weights corresponding to the positive and negative samples respectively, thus obtaining the sub-loss value corresponding to the text binary classifier.

[0093] In some embodiments, the angle feature loss function further includes a hard sample mining parameter. The method also includes determining the weights corresponding to positive and negative samples based on the positive / negative sample balance parameter and the hard sample mining parameter in the angle feature loss function. It is understood that the loss value for hard samples is higher and more sensitive than that for simple samples; therefore, the hard sample mining parameter r is introduced. Since the weight of each sample text in both positive and negative samples takes into account the hard sample mining parameter r, the text binary classifier can pay more attention to hard samples, thereby improving overall convergence and generalization.

[0094] Step 112: Determine the target loss value based on the sub-loss value corresponding to each text binary classifier to be trained, and iteratively optimize the training in the direction of minimizing the target loss value to obtain the trained text multi-classification model.

[0095] Multiple text binary classifiers obtain corresponding sub-loss values ​​through steps 106 to 110. The computer device can determine a total target loss value based on the sub-loss values ​​corresponding to each text binary classifier to be trained. This target loss value is the overall loss value of the text multi-classification model. The training is iteratively optimized in the direction of minimizing the target loss value. The trained text multi-classification model is then subjected to parameter tuning and iterative model optimization training. After the training termination condition is met, the trained text multi-classification model is obtained.

[0096] It is understandable that optimization training can be done in multiple rounds. In each round of training, steps 106 to 112 can be executed to adjust the model parameters. After completing one round of training, the next round can be executed iteratively until the training termination condition is met.

[0097] It should be noted that since the target loss value is calculated by substituting the angle between the text feature vector and the classifier weight vector into the angle feature loss function, the target loss value is minimized by minimizing the angle between the text feature vector and the classifier weight vector, thereby achieving model optimization training.

[0098] In the above embodiments, each sample text is input into the text multi-classification model to be trained to extract the text feature vector of each sample text; for each text binary classifier in the text multi-classification model to be trained, sample texts belonging to the corresponding target category are taken as positive samples, and sample texts not belonging to the target category are taken as negative samples, and the classifier weight vector of the text binary classifier and the text feature vectors corresponding to the positive and negative samples are normalized to a unit hypersphere; an angular feature loss function is determined; the angular feature loss function includes a positive and negative sample balance parameter; the angle between the normalized text feature vectors of the positive and negative samples and the classifier weight vector is substituted into the angular feature loss function to obtain the sub-loss value corresponding to the text binary classifier; based on the sub-loss value corresponding to each text binary classifier to be trained, a target loss value is determined, and the training is iteratively optimized in the direction of minimizing the target loss value to obtain the trained text multi-classification model. By transforming a multi-class classification task into multiple binary classification tasks and normalizing the text feature vectors and classifier weights onto a hypersphere, and performing classification training on a unit hypersphere based on an angle feature loss function, the classifier can learn more discriminative angle features on the hypersphere, thereby enhancing the text binary classifier's ability to distinguish text. Furthermore, by optimizing the loss function based on a positive-negative sample balancing parameter, the training process pays more attention to the minority of positive samples, achieving a positive-negative sample balancing adjustment and further improving the model's classification accuracy.

[0099] In some embodiments, the angle feature loss function further includes a similarity calculation function constructed based on the angle variable. In this embodiment, step 110 includes: substituting the angle between the text feature vector normalized to a unit hypersphere and the classifier weight vector into the similarity calculation function to calculate the cosine similarity corresponding to each positive and negative sample; determining the loss value corresponding to each positive and negative sample based on the cosine similarity corresponding to each positive and negative sample; and obtaining the sub-loss value corresponding to the text binary classifier based on the loss value corresponding to each positive and negative sample and the positive and negative sample balance parameter.

[0100] The similarity calculation function is used to calculate the cosine similarity between the text feature vector normalized to a unit hypersphere and the classifier weight vector. The angle variable is used to represent the angle between the text feature vector and the classifier weight vector.

[0101] For example, the computer device can determine the angle between the normalized text feature vectors and classifier weight vectors of each positive and negative sample. When calculating the loss value for each sample text in the positive and negative samples, the angle between the normalized text feature vector and the classifier weight vector of that sample text is assigned to the angle variable in the similarity calculation function to calculate the cosine similarity of that sample text. Then, the loss value for that sample text is determined based on the cosine similarity. It can be understood that after performing the above processing for each sample text in the positive and negative samples, the loss value corresponding to each positive and negative sample can be obtained. The computer device can then obtain the sub-loss value corresponding to the text binary classifier based on the loss values ​​corresponding to the positive and negative samples and the positive and negative sample balance parameters. For example, the computer device can weight the loss values ​​corresponding to the positive and negative samples according to the weights corresponding to the positive and negative samples to obtain the sub-loss value corresponding to the text binary classifier. The weights corresponding to the positive and negative samples are determined based on the positive and negative sample balance parameters.

[0102] For example, the cosine similarity between positive and negative samples can be cosine similarity. The formula for the angular feature loss function in multi-class classification is as follows:

[0103]

[0104] Where λ is the positive and negative sample balance parameter, λ∈[0,1]; θ is the angle between the classifier weight vector and the text feature vector. y θ is the angle between the classifier weight vector and the text feature vector of positive samples belonging to the target class y; i θ is the angle between the classifier weight vector and the text feature vector of a negative sample belonging to a non-target class i, where i is a preset class other than the target class y from K preset classes. The K preset classes are the classes that the text multi-classification model needs to be able to recognize. cos(θ) is the cosine similarity; cos(θ) y ) is the cosine similarity between positive samples, cos(θ) i ) is the cosine similarity corresponding to the negative samples. log(1+exp(-cos(θy))) is a subfunction used to calculate the loss value of the positive samples; It is a subfunction used to calculate the loss value of negative samples.

[0105] In the above embodiments, on the unit hypersphere, the cosine similarity is calculated based on the angle to determine the loss value, which is equivalent to changing the model optimization objective to the minimum angle, making the training process more convenient and the training results more accurate, thereby improving efficiency and classification accuracy.

[0106] In some embodiments, the angular feature loss function includes a first corner margin parameter and a second corner margin hyperparameter. In this embodiment, determining the loss value corresponding to each positive and negative sample based on the cosine similarity of each sample includes: for positive samples, determining the loss value corresponding to the positive sample based on the cosine similarity and the first corner margin parameter; for negative samples, determining the loss value corresponding to the negative sample based on the cosine similarity and the second corner margin hyperparameter.

[0107] The first corner margin parameter controls the corner margin size of positive samples, while the second corner margin hyperparameter controls the corner margin size of negative samples. Specifically, the first corner margin parameter continuously reduces the corner margin between positive samples, thus decreasing the intra-class difference between positive samples; the second corner margin hyperparameter continuously reduces the corner margin between negative samples, thus decreasing the intra-class difference between negative samples. This, in turn, continuously increases the inter-class difference, resulting in a more separated decision region and better classification performance.

[0108] For example, when calculating the loss value for positive samples, the computer device can consider a first corner margin parameter to determine the loss value for the positive sample based on the cosine similarity and the first corner margin parameter. When calculating the loss value for negative samples, a second corner margin hyperparameter is considered to determine the loss value for the negative sample based on the cosine similarity and the second corner margin hyperparameter. Because the respective corner margin hyperparameters are considered when calculating the loss values ​​for positive and negative samples, the intra-class discrepancy can be continuously reduced during model training to optimize the model's classification ability.

[0109] In some embodiments, because the text binary classifier has a fixed classification threshold, too many negative samples are compressed into a fixed angular space. This leads to training difficulties because negative samples also have different categories. Forcibly compressing these negative sample data is equivalent to compressing data of different categories into a certain region, which conflicts with the goal of optimizing classification performance. Therefore, a bias hyperparameter b can be added to the angular feature loss function to change the decision boundary, which is equivalent to optimizing the angular margin, thereby compressing the positive sample space and relaxing the spatial constraints of the negative samples.

[0110] In some embodiments, the angle feature loss function is as follows:

[0111]

[0112] Where, m p This is the first angle margin parameter, m n This is the second corner margin parameter. y It is a bias hyperparameter set for positive samples, b iThese are the bias hyperparameters set for negative samples. λ is the positive-negative sample balancing parameter, and r is the hard sample mining parameter.

[0113] In some embodiments, m p =m n =0.4.

[0114] To facilitate understanding, we will now combine... Figure 2 Let's illustrate with examples. Figure 2 This is a schematic diagram illustrating the compression of the positive sample space and the relaxation of the negative sample space constraints in one embodiment. It should be understood that because in a text binary classifier, only the sample text of the target class is a positive sample, while the sample text of other classes is a negative sample, the space occupied by negative samples should theoretically be much larger than that of the positive class. However, since positive samples in a text binary classifier have a fixed classification threshold, this leads to… Figure 2 As shown in (a), the space occupied by positive samples is the same as that occupied by negative samples, resulting in many negative samples being compressed into a fixed angular space. Therefore, a bias hyperparameter b can be added to change the decision boundary. Figure 2 In equation (a), the decision boundary is r·cos(θ)=0. By adding the bias hyperparameter b, the decision boundary changes to r·cos(θ)+b=0. At this point, the corner margin of the positive class sample is... The corner margins of the negative class samples are That is, such as Figure 2 As shown in (b), the positive sample space is compressed while the negative sample space is relaxed.

[0115] In some embodiments, the angle feature loss function further includes a similarity mapping function. In this embodiment, determining the loss value corresponding to each positive and negative sample based on the cosine similarity of each positive and negative sample includes: mapping the calculated cosine similarity through the similarity mapping function to obtain the mapping results corresponding to each positive and negative sample; and determining the loss value corresponding to each positive and negative sample based on the mapping results corresponding to each positive and negative sample.

[0116] The similarity mapping function contains a nested similarity calculation function, meaning the similarity calculation function is a subfunction of the similarity mapping function. The similarity mapping function maps the cosine similarity calculated by the similarity calculation function to a larger range of values, thus making the differences between sample pairs with different cosine similarities more significant.

[0117] For example, after the similarity calculation function calculates the cosine similarity between positive and negative samples based on the angle between the text feature vector and the classifier weight vector, the computer device can map the calculated cosine similarity using a similarity mapping function to obtain the mapping results corresponding to the positive and negative samples respectively. Then, based on the mapping results corresponding to the positive and negative samples respectively, the loss values ​​corresponding to the positive and negative samples can be determined.

[0118] In some embodiments, the similarity mapping function further includes a similarity adjustment parameter. This parameter constrains the overlap between the cosine similarity distributions corresponding to positive and negative samples, preventing excessive overlap. In other words, the similarity adjustment parameter reduces the overlap between the cosine similarity distributions corresponding to positive and negative samples, thus contributing to improved classification accuracy.

[0119] In some embodiments, the formula for the angle feature loss function is as follows:

[0120]

[0121] Here, g(cos(θ)) is a similarity mapping function used to map the cosine similarity calculated by the cosine similarity function.

[0122] In some embodiments, the specific expression of the similarity mapping function is as follows:

[0123]

[0124] Where z represents cos(θ), which is the cosine similarity; t is the similarity adjustment parameter.

[0125] Figure 3 This is a schematic diagram illustrating the principle of a text classification method in one embodiment. See also... Figure 3 The computer equipment first preprocesses the sample text set (i.e., data preprocessing), and then encodes the sample text based on the BERT model (i.e., BERT text encoding) to obtain text feature vectors. Next, it normalizes the text feature vectors and the classifier weight vectors of the text binary classifier to constrain the classification process to a unit hypersphere (i.e., a hypersphere classifier). Then, based on the unit hypersphere, the multi-classification task is transformed into multiple binary classification tasks. Using hypersphere optimization strategies such as positive / negative sample balancing, hard sample mining, corner margin adjustment, and similarity adjustment, the loss function is optimized, thereby training a more accurate text multi-classification model.

[0126] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0127] Based on the same inventive concept, this application also provides a text classification apparatus for implementing the text classification method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more text classification apparatus embodiments provided below can be found in the limitations of the text classification method described above, and will not be repeated here.

[0128] In one embodiment, such as Figure 4 As shown, a text classification device is provided, including: a feature extraction module 402, a normalization module 404, and an optimization module 406, wherein:

[0129] The feature extraction module 402 is used to obtain a sample text set; the sample text set includes sample texts of multiple preset categories; each sample text is input into the text multi-classification model to be trained to extract the text feature vector of each sample text; the text multi-classification model to be trained includes multiple text binary classifiers to be trained; each text binary classifier corresponds to one of the target categories of multiple preset categories.

[0130] The normalization module 404 is used to normalize the classifier weight vector of the text binary classifier and the text feature vectors corresponding to the positive and negative samples to the unit hypersphere for each text binary classifier.

[0131] The optimization module 406 is used to determine the angle feature loss function. The angle feature loss function includes a positive and negative sample balance parameter. The angle between the normalized text feature vectors of the positive and negative samples and the classifier weight vector is substituted into the angle feature loss function to obtain the sub-loss value corresponding to the text binary classifier. Based on the sub-loss value corresponding to each text binary classifier to be trained, the target loss value is determined, and the training is iteratively optimized in the direction of minimizing the target loss value to obtain the trained text multi-classification model.

[0132] In some embodiments, the angle feature loss function further includes a similarity calculation function constructed based on the angle variable; the similarity calculation function is used to calculate the cosine similarity between the text feature vector normalized to a unit hypersphere and the classifier weight vector; the angle variable is used to characterize the angle between the text feature vector and the classifier weight vector. The optimization module 406 is further used to substitute the angle between the text feature vector normalized to a unit hypersphere and the classifier weight vector into the similarity calculation function to calculate the cosine similarity corresponding to each positive and negative sample; determine the loss value corresponding to each positive and negative sample based on the cosine similarity corresponding to each positive and negative sample; and obtain the sub-loss value corresponding to the text binary classifier based on the loss value corresponding to each positive and negative sample and the positive and negative sample balance parameter.

[0133] In some embodiments, the angle feature loss function further includes a similarity mapping function; the similarity calculation function is a subfunction of the similarity mapping function. The optimization module 406 is also used to map the calculated cosine similarity through the similarity mapping function to obtain the mapping results corresponding to the positive and negative samples respectively; based on the mapping results corresponding to the positive and negative samples respectively, the loss values ​​corresponding to the positive and negative samples are determined respectively.

[0134] In some embodiments, the similarity mapping function further includes a similarity adjustment parameter; the similarity adjustment parameter is used to constrain the degree of overlap between the cosine similarity distributions corresponding to positive and negative samples, respectively.

[0135] In some embodiments, the angular feature loss function includes a first corner margin parameter and a second corner margin hyperparameter; the first corner margin parameter controls the corner margin size of positive samples; and the second corner margin hyperparameter controls the corner margin size of negative samples. The optimization module 406 is further configured to determine the loss value corresponding to a positive sample based on the cosine similarity and the first corner margin parameter; and to determine the loss value corresponding to a negative sample based on the cosine similarity and the second corner margin hyperparameter.

[0136] In some embodiments, the optimization module 406 is further configured to substitute the angle between the normalized text feature vectors and classifier weight vectors corresponding to the positive and negative samples into the angle feature loss function to obtain the loss values ​​corresponding to the positive and negative samples respectively; and to weight the loss values ​​corresponding to the positive and negative samples according to the weights corresponding to the positive and negative samples respectively to obtain the sub-loss values ​​corresponding to the text binary classifier; wherein, the weights corresponding to the positive and negative samples are determined according to the positive and negative sample balance parameter in the angle feature loss function; the weight corresponding to the positive sample is greater than the weight corresponding to the negative sample.

[0137] In some embodiments, the angle feature loss function further includes a hard sample mining parameter. The optimization module 406 is also used to determine the weights corresponding to the positive and negative samples based on the positive and negative sample balance parameter and the hard sample mining parameter in the angle feature loss function.

[0138] Each module in the aforementioned text classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0139] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores sample text sets. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a text classification method.

[0140] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a text classification method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0141] Those skilled in the art will understand that Figure 5 and Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above embodiments.

[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above embodiments.

[0144] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above embodiments.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A text classification method, characterized in that, The method includes: Obtain a sample text set; the sample text set includes sample texts of multiple preset categories; Each of the sample texts is input into a text multi-classification model to be trained to extract the text feature vector of each sample text; the text multi-classification model to be trained includes multiple text binary classifiers to be trained; each text binary classifier corresponds to one of the target categories of the multiple preset categories; For each of the text binary classifiers, sample texts belonging to the corresponding target category are taken as positive samples, and sample texts not belonging to the target category are taken as negative samples. The classifier weight vector of the text binary classifier and the text feature vectors corresponding to the positive and negative samples are normalized to the unit hypersphere. Determine the angle feature loss function; the angle feature loss function includes a positive and negative sample balance parameter; Substitute the angles between the normalized text feature vectors of the positive and negative samples and the classifier weight vectors into the angle feature loss function to obtain the sub-loss values ​​corresponding to the text binary classifier. The target loss value is determined based on the sub-loss value corresponding to each text binary classifier to be trained, and the training is iteratively optimized in the direction of minimizing the target loss value to obtain the trained text multi-classification model.

2. The method according to claim 1, characterized in that, The angle feature loss function also includes a similarity calculation function constructed based on the angle variable; the similarity calculation function is used to calculate the cosine similarity between the text feature vector normalized to a unit hypersphere and the classifier weight vector; the angle variable is used to characterize the angle between the text feature vector and the classifier weight vector; The step of substituting the angles between the normalized text feature vectors of positive and negative samples and the classifier weight vector into the angle feature loss function to obtain the sub-loss value corresponding to the text binary classifier includes: The angle between the text feature vector normalized to a unit hypersphere and the classifier weight vector is substituted into the similarity calculation function to calculate the cosine similarity corresponding to each positive and negative sample. Based on the cosine similarity of each positive and negative sample, determine the loss value corresponding to each positive and negative sample. Based on the loss values ​​corresponding to the positive and negative samples and the balance parameters of the positive and negative samples, the sub-loss values ​​corresponding to the text binary classifier are obtained.

3. The method according to claim 2, characterized in that, The angle feature loss function further includes a similarity mapping function; the similarity calculation function is a sub-function of the similarity mapping function; The step of determining the loss value corresponding to each positive and negative sample based on the cosine similarity of each positive and negative sample includes: The calculated cosine similarity is mapped using the similarity mapping function to obtain the mapping results corresponding to the positive and negative samples, respectively. Based on the mapping results corresponding to the positive and negative samples respectively, the loss values ​​corresponding to the positive and negative samples are determined respectively.

4. The method according to claim 3, characterized in that, The similarity mapping function also includes a similarity adjustment parameter; the similarity adjustment parameter is used to constrain the degree of overlap between the cosine similarity distributions corresponding to positive and negative samples respectively.

5. The method according to claim 2, characterized in that, The angle feature loss function includes a first angle margin parameter and a second angle margin hyperparameter; the first angle margin parameter is used to control the angle margin size of positive samples; the second angle margin hyperparameter is used to control the angle margin size of negative samples. The step of determining the loss value corresponding to each positive and negative sample based on the cosine similarity of each positive and negative sample includes: For positive samples, the loss value corresponding to the positive sample is determined based on the cosine similarity of the positive sample and the first corner margin parameter; For negative samples, the loss value corresponding to the negative sample is determined based on the cosine similarity of the negative sample and the second angle margin hyperparameter.

6. The method according to claim 1, characterized in that, The step of substituting the angles between the normalized text feature vectors of positive and negative samples and the classifier weight vector into the angle feature loss function to obtain the sub-loss value corresponding to the text binary classifier includes: Substitute the angle between the normalized text feature vector and the classifier weight vector corresponding to the positive and negative samples into the angle feature loss function to obtain the loss values ​​corresponding to the positive and negative samples respectively. The loss values ​​corresponding to the positive and negative samples are weighted according to the weights corresponding to the positive and negative samples respectively to obtain the sub-loss values ​​corresponding to the text binary classifier; The weights corresponding to the positive and negative samples are determined based on the positive and negative sample balance parameter in the angle feature loss function; the weight of the positive sample is greater than the weight of the negative sample.

7. The method according to claim 6, characterized in that, The angle feature loss function also includes hard sample mining parameters; The method further includes: The weights corresponding to the positive and negative samples are determined based on the positive and negative sample balance parameters and the hard sample mining parameters in the angle feature loss function.

8. A text classification device, characterized in that, The device includes: A feature extraction module is used to acquire a sample text set; the sample text set includes sample texts of multiple preset categories; each sample text is input into a text multi-classification model to be trained to extract the text feature vector of each sample text; the text multi-classification model to be trained includes multiple text binary classifiers to be trained; each text binary classifier corresponds to one of the target categories of the multiple preset categories; The normalization module is used to, for each of the text binary classifiers, take the sample text belonging to the corresponding target category as positive samples and the sample text not belonging to the target category as negative samples, and normalize the classifier weight vector of the text binary classifier and the text feature vectors corresponding to the positive and negative samples to the unit hypersphere. An optimization module is used to determine an angle feature loss function, which includes a positive and negative sample balance parameter. The angle between each of the normalized text feature vectors of the positive and negative samples and the classifier weight vector is substituted into the angle feature loss function to obtain the sub-loss value corresponding to the text binary classifier. Based on the sub-loss value corresponding to each text binary classifier to be trained, a target loss value is determined, and the training is iteratively optimized in the direction of minimizing the target loss value to obtain the trained text multi-classification model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images