Text generalized intention discovery method based on deviation self-correction calibration and related device

Abstract

The invention discloses a text generalized intention discovery method and system based on deviation self-correction calibration. The text generalized intention discovery method comprises the steps of obtaining a prediction text sample and inputting the prediction text sample into a preset generalized intention discovery model to obtain model original output; and classifying the original output of the model by adopting a Softmax function, and taking the category to which the maximum Softmax value belongs as the sample prediction category of the text sample. Wherein the generalized intention discovery model is internally provided with a biased branch and a trainable branch, firstly, a biased branch with deviation is obtained through pre-training, and model parameters are fixed; then, respectively inputting a training text sample into a pre-trained biased branch and a pre-trained trainable branch, outputting two original outputs, adjusting the original output of the trainable branch by utilizing the original output of the biased branch, and relieving category deviation and category confusion by utilizing the deviation of a model to a known category; the method plays a role in relieving category deviation and category confusion, and effectively improves the recognition accuracy of the model to the new category.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and relates to a method for discovering generalized intent of text based on deviation self-correction calibration and a related device. Background Art

[0002] Since many current deep learning models are trained on closed datasets, they may misidentify new intent categories. For example, in the fields of intent discovery and image classification, these trained models may encounter unlabeled training text samples that belong to categories that have not been seen before.

[0003] In order to solve the problems existing in such open scenarios, the academic community has proposed generalized intent discovery, which aims to identify known intent categories and unknown intent categories from unlabeled training text samples. The current mainstream method of generalized intent discovery is to tailor or customize the pre-trained model on labeled training text samples for specific tasks. However, in the absence of labeled data for new intent categories, these pre-trained models will show a strong bias towards known intent categories, resulting in the misclassification of samples of new intent categories as known intent categories.

[0004] In order to alleviate the bias of existing methods towards known intent categories, existing methods fine-tune the pre-trained model on unlabeled training text samples through pseudo-label training, contrastive learning, and transfer learning. Although the effects of the above methods have been improved to a certain extent, these methods still have certain problems: (1) The model still has bias towards known intent categories, which will cause new intent categories to be misclassified as known intent categories. (2) The model lacks effective supervision for new intent categories, which will cause new intent categories to be misclassified as other new intent categories. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a text generalized intent discovery method and related devices based on deviation self-correction calibration.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for discovering generalized intent of text based on deviation self-correction calibration, comprising:

[0008] Obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original output of the model;

[0009] The Softmax function is used to classify the original output of the model, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample;

[0010] Among them, the generalized intent discovery model is obtained in the following way:

[0011] Construct biased branches and trainable branches; wherein the biased branches include sequentially connected pre-trained BERT models and pre-trained fully connected layers, and the trainable branches include sequentially connected BERT models and fully connected layers;

[0012] The parameters of the biased branch are fixed, and the parameters of the BERT model are initialized with the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized with the category prototype;

[0013] For each training text sample, a biased output L is obtained through a biased branch. bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, a migration matrix is ​​constructed and normalized to obtain a normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L that alleviates category confusion CCM ; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ];

[0014] According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And the true label Y of the labeled training text samples is calculated by the cross entropy loss function. l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sip and l cont Perform weighted combination to obtain the final loss l;

[0015] According to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

[0016] Optionally, the parameters of the fully connected layer initialized using the category prototype include:

[0017] For labeled training text samples, the category prototype is calculated by the following formula:

[0018]

[0019] in, represents the prototype of category j, x i is a training text sample; f θ is the BERT model; C j are all training text samples belonging to category j, and M is the number of labeled training text sample categories;

[0020] For the unlabeled training text samples, KMeans clustering is first performed to obtain a set of all categories. K is the sum of the number of known categories and the number of unknown categories;

[0021] For unlabeled training text samples, the category prototype is calculated by the following formula:

[0022]

[0023] in,

[0024] Align the category prototypes of the unlabeled training text samples with the category prototypes of the labeled training text samples, and obtain the optimal arrangement of the category prototypes of the unlabeled training text samples by calculating the distance between the arrangement of the category prototypes of all unlabeled training text samples and the arrangement of the category prototypes of the labeled training text samples.

[0025]

[0026] in, Represents the calculation of the Euclidean distance between two vectors;

[0027] Will Normalize and get the fully connected layer g φ Initial value of:

[0028] g φ =[c 1 ,...,c M ,...,c K ]

[0029] Among them, the elements indexed from 1 to M represent the category prototypes of labeled training text samples.

[0030] Optionally, the biased output L is obtained by the biased branch bias , the training output L is obtained through the trainable branch θ , combined with a biased output weight factor α i Get the output L that alleviates the category biasCBM include:

[0031] Calculate the training text sample x i Entropy E i :

[0032]

[0033] Among them, p ij =Softmax(L bias (x i )) is the training text sample x i Probability distribution on biased branches;

[0034] According to the training text sample x i Entropy E i , the biased output weight factor α is obtained by the following formula = {α i}:

[0035] α i =β·Sigmoid(E i -E max )

[0036] Among them, β is an adjustable weight, E max is the maximum entropy value in the current training text sample batch;

[0037] The output L that alleviates the category bias is obtained by the following formula CBM :

[0038] L CBM =L θ [:M]-α·L bias

[0039] Among them, M represents the number of known categories.

[0040] Optionally, the step of constructing a migration matrix according to the category prototype and normalizing it to obtain a normalized migration matrix, and outputting the biased L according to the normalized migration matrix bias and training output L θ Get the output L that alleviates category confusion CCM include:

[0041] Construct the migration matrix T according to the category prototype M×n :

[0042]

[0043] Among them, {c 1 ,...,c M} and {c M+1 ,...,c M+N} are prototypes of known and unknown categories;

[0044] Normalize each row in the migration matrix to obtain the transition probability from each known category to the unknown category;

[0045] The output L that alleviates category confusion is obtained by the following formula CCM :

[0046] L CCM =L θ [M:]+α·L bias T ×

[0047] Among them, L θ [M:] represents L θ The last N elements of correspond to N unknown categories.

[0048] Optionally, the L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm include:

[0049] The pseudo labels of unlabeled training text samples are obtained by the following formula

[0050]

[0051]

[0052] Among them, Tr represents the rank of the matrix, ∈ is the weight factor, E represents the entropy function, B is the batch size, K is the number of categories, and vector 1 B and 1 K are unit vectors of dimension B and K respectively.

[0053] Optionally, the true labels Y of the labeled training text samples are calculated by the cross entropy loss function. l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l uont The final loss l obtained by weighted combination includes:

[0054] The true labels Y of the labeled training text samples are calculated by the cross entropy loss function l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup :

[0055]

[0056] Among them, λ 1 is an adjustable weight, and are the categories of labeled training text samples and unlabeled training text samples predicted by the original classifier, respectively, l The true category of the labeled training text samples;

[0057] Compute the contrastive loss for labeled training text samples via instance-level contrastive learning

[0058]

[0059] in, is a set of labeled training text samples, are all training text samples with the same label in the same batch; τ is a preset constant variable;

[0060] Computing the contrastive loss for unlabeled training text samples via instance-level contrastive learning

[0061]

[0062] in, is a set of unlabeled training text samples, represents the enhanced training text samples;

[0063] Contrasting loss l between labeled training text samples and unlabeled training text samples cont :

[0064]

[0065] Yes sup and l cont Perform weighted combination to get the final loss l:

[0066]

[0067] Among them, λ 2 yes The preset weights.

[0068] Optionally, the method of using the Softmax function to classify the original output of the model and taking the category to which the Softmax maximum value belongs as the sample prediction category of the text sample includes:

[0069] For the predicted text sample x, calculate the model's original output L:

[0070] L = g φ (f θ (x))

[0071] Among them, L is a K-dimensional vector, K represents the total number of categories; g φ is the fully connected layer of the generalized intent discovery model, f θ BERT model for generalized intent discovery model;

[0072] Use the Softmax function to classify L, and predict the category of the text sample x to be the category to which the Softmax maximum value belongs:

[0073]

[0074] Among them, y is the predicted category of the predicted text sample x.

[0075] In a second aspect, the present invention provides a text generalized intent discovery system based on deviation self-correction calibration, comprising:

[0076] The text sample model processing module is used to obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original output of the model;

[0077] The text sample category prediction module is used to classify the original output of the model using the Softmax function, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample;

[0078] Among them, the generalized intent discovery model is obtained in the following way:

[0079] Construct biased branches and trainable branches; wherein the biased branches include sequentially connected pre-trained BERT models and pre-trained fully connected layers, and the trainable branches include sequentially connected BERT models and fully connected layers;

[0080] The parameters of the biased branch are fixed, and the parameters of the BERT model are initialized with the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized with the category prototype;

[0081] For each training text sample, a biased output L is obtained through a biased branch. bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category biasCBM ; According to the category prototype, a migration matrix is ​​constructed and normalized to obtain a normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L that alleviates category confusion CCM ; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ];

[0082] According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And the true label Y of the labeled training text samples is calculated by the cross entropy loss function. l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont Perform weighted combination to obtain the final loss l;

[0083] According to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

[0084] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for discovering generalized intent of text based on deviation self-correction calibration when executing the computer program.

[0085] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for discovering generalized intent of text based on deviation self-correction calibration are implemented.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] The present invention is based on a method for discovering generalized intent of text based on self-correction calibration of deviations. The original output of the model is obtained by obtaining a predicted text sample and inputting it into a preset generalized intent discovery model. Then, the original output of the model is classified by using the Softmax function, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample, so as to realize the effective recognition of the new intent category of the predicted text sample. Among them, a biased branch and a trainable branch are set inside the generalized intent discovery model. First, a biased branch with a bias is obtained by pre-training, and the model parameters are fixed. Then, the training text sample is input into the pre-trained biased branch and the trainable branch respectively, and two original outputs are output. The original output of the biased branch is used to adjust the original output of the trainable branch, and the deviation of the model to the known category is used to alleviate the category deviation and category confusion; in addition, the present invention proposes a method for adjusting the output of the biased model, which has a role in alleviating both the category deviation and the category confusion, and effectively improves the recognition accuracy of the model for new categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 This is a flow chart of a method for discovering generalized intent in text based on deviation self-correction calibration according to an embodiment of the present invention.

[0089] Figure 2 Schematic diagram of the training principle of the generalized intent discovery model according to an embodiment of the present invention.

[0090] Figure 3 This is a structural block diagram of a text generalized intent discovery system based on deviation self-correction calibration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0092] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0093] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0094] See also Figure 1 In one embodiment of the present invention, a method for discovering generalized intent in text based on deviation self-correction calibration is provided to improve the recognition accuracy of new categories of text samples.

[0095] Specifically, the method for discovering generalized intent of text based on deviation self-correction calibration of the present invention comprises the following steps:

[0096] S1: Obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original output of the model.

[0097] S2: Use the Softmax function to classify the original output of the model, and take the category to which the Softmax maximum value belongs as the sample prediction category of the text sample.

[0098] The generalized intent discovery model is obtained by: constructing a biased branch and a trainable branch; wherein the biased branch includes a pre-trained BERT model and a pre-trained fully connected layer connected in sequence, and the trainable branch includes a BERT model and a fully connected layer connected in sequence; fixing the parameters of the biased branch, and using the parameters of the pre-trained BERT model to initialize the parameters of the BERT model, and using the category prototype to initialize the parameters of the fully connected layer; for each training text sample, a biased output L is obtained through the biased branch. bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, a migration matrix is ​​constructed and normalized to obtain a normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L that alleviates category confusion CCM ; Merge L CBM and L CCMGet the deviation self-calibration output L c =[L CBM ,L CCM ]; According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And the true label Y of the labeled training text samples is calculated by the cross entropy loss function. l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont A weighted combination is performed to obtain the final loss l; according to l, the parameters of the trainable branches are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

[0099] The present invention is based on a method for discovering generalized intent of text based on self-correction calibration of deviations. The original output of the model is obtained by obtaining a predicted text sample and inputting it into a preset generalized intent discovery model. Then, the original output of the model is classified by using the Softmax function, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample, so as to realize the effective recognition of the new intent category of the predicted text sample. Among them, a biased branch and a trainable branch are set inside the generalized intent discovery model. First, a biased branch with a bias is obtained by pre-training, and the model parameters are fixed. Then, the training text sample is input into the pre-trained biased branch and the trainable branch respectively, and two original outputs are output. The original output of the biased branch is used to adjust the original output of the trainable branch, and the deviation of the model to the known category is used to alleviate the category deviation and category confusion; in addition, the present invention proposes a method for adjusting the output of the biased model, which has a role in alleviating both the category deviation and the category confusion, and effectively improves the recognition accuracy of the model for new categories.

[0100] In one possible implementation, the training of the generalized intent discovery model includes the following detailed steps:

[0101] 1. Obtain biased model f through pre-training bias and biased classifier g bias , where there is a biased model f biasTo pre-train the BERT model, the biased classifier is a pre-trained fully connected layer. In pre-training, the cross entropy loss function is used for labeled training text samples, and the MLM (masked language modeling) loss function is used for unlabeled training text samples. The biased model f bias : and biased classifier g bias : Among them, d is the feature dimension and M is the number of known categories.

[0102] 2. Freeze the pre-trained biased model f bias and biased classifier g bias Parameters. Using a trainable model f θ With a trainable classifier g φ Used together for training the entire model. θ is with f bias BERT model with the same structure, g φ is a fully connected layer, θ and φ are learnable parameters. bias The parameters to initialize f θ , use the category prototype to initialize the classifier g φ .

[0103] The method for calculating the category prototype is as follows: the KMeans clustering algorithm is used on the unlabeled training text samples, and each category center obtained by KMeans is regarded as a category prototype. Then, an alignment algorithm is used to align the known categories in g bias and g φ Then, g φ Initialized to l 2 Normalized aligned category prototype: g φ =[c 1 ,...,c M ,...,c K ], where indexes 1 to M represent known category prototypes.

[0104] 3. For each training text sample x i , first input into the pre-trained biased model f bias and biased classifier g bias , and get the biased output L bias ; Then input to the trainable model f θ and a trainable classifier g φ In the output L θ . Calculate the biased output weight factor α i . Use L θ Subtract α·L bias, and obtain the output L that reduces the model category deviation CBM .

[0105] 4. Use the category prototype to measure the similarity between categories and calculate the migration matrix T based on the category prototype M×N , M and N represent the number of known categories and unknown categories respectively. Then, the migration matrix T M×N Normalize to obtain the transition probability from each known category to the new category. According to the migration matrix T, the output L after alleviating the category confusion is obtained. CCM Combine the output of mitigating category deviation and the output of mitigating category confusion to obtain the final deviation self-calibration output L c =[L CBM ,L CCM ].

[0106] 5. According to the calibrated output L c , the pseudo labels of the unlabeled training text samples are obtained through the SK algorithm. The true labels Y of the labeled training text samples are calculated through the cross entropy loss function. l With predicted labels The loss between the unlabeled training text samples and the pseudo labels With predicted labels The loss between them is weighted combination to get the total cross entropy loss l sup . Use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont . sup and l cont The final loss value l is obtained by weighted combination. The model parameters are continuously updated through back propagation until l converges, and the trained model f is obtained. θ and classifier g φ , that is, the generalized intent discovery model.

[0107] In a possible implementation, the method of initializing the parameters of the fully connected layer using the category prototype includes:

[0108] For labeled training text samples, the category prototype is calculated by the following formula:

[0109]

[0110] in, represents the prototype of category j, x i is a training text sample; f θ is the BERT model; C j are all training text samples belonging to category j, and M is the number of labeled training text sample categories.

[0111] For the unlabeled training text samples, KMeans clustering is first performed to obtain a set of all categories. K is the sum of the number of known categories and the number of unknown categories.

[0112] For unlabeled training text samples, the category prototype is calculated by the following formula:

[0113]

[0114] in,

[0115] Align the category prototypes of the unlabeled training text samples with the category prototypes of the labeled training text samples, and obtain the optimal arrangement of the category prototypes of the unlabeled training text samples by calculating the distance between the arrangement of the category prototypes of all unlabeled training text samples and the arrangement of the category prototypes of the labeled training text samples.

[0116]

[0117] in, Represents the calculation of the Euclidean distance between two vectors.

[0118] Will Normalize and get the fully connected layer g φ Initial value of:

[0119] g φ =[c 1 ,...,c M ,...,c K ]

[0120] Among them, the elements indexed from 1 to M represent the category prototypes of labeled training text samples.

[0121] In a possible implementation manner, the biased output L is obtained by the biased branch. bias , the training output L is obtained through the trainable branch θ , combined with a biased output weight factor α i Get the output L that alleviates the category bias CBM Includes: Calculate training text sample x i Entropy E i :

[0122]

[0123] Among them, p ij =Softmax(L bias (x i )) is the training text sample x i Probability distribution over biased branches.

[0124] According to the training text sample x i Entropy E i , the biased output weight factor α is obtained by the following formula = {α i}:

[0125] α i =β·Sigmoid(E i -E max )

[0126] Among them, β is an adjustable weight, E max is the maximum entropy value in the current training text sample batch.

[0127] The output L that alleviates the category bias is obtained by the following formula CBM :

[0128] L CBM =L θ [:M]-α·L bias

[0129] Among them, M represents the number of known categories.

[0130] In a possible implementation, the step of constructing a migration matrix according to the category prototype and normalizing the matrix to obtain a normalized migration matrix, and outputting the normalized migration matrix and the biased output L bias and training output L θ Get the output L that alleviates category confusion CCM Includes: Constructing the migration matrix T according to the category prototype M×N :

[0131]

[0132] Among them, {c 1 ,...,c M} and {c M+1 ,...,c M+N} are prototypes of known and unknown categories;

[0133] Normalize each row in the migration matrix to obtain the transition probability from each known category to the unknown category;

[0134] The output L that alleviates category confusion is obtained by the following formula CCM :

[0135] L CCM =L θ [M:]+α·L bias T ×T

[0136] Among them, L θ [M:] represents Lθ The last N elements of correspond to N unknown categories.

[0137] In a possible implementation manner, the c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm Including: Get the pseudo label of the unlabeled training text sample by the following formula

[0138]

[0139]

[0140] Among them, Tr represents the rank of the matrix, ∈ is the weight factor, E represents the entropy function, B is the batch size, K is the number of categories, and vector 1 B and 1 K are unit vectors of dimension B and K respectively.

[0141] In a possible implementation, the cross entropy loss function is used to calculate the true labels Y of the labeled training text samples. l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples xont ; for l sup and l cont The final loss l obtained by weighted combination includes:

[0142] The true labels Y of the labeled training text samples are calculated by the cross entropy loss function l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup :

[0143]

[0144] Among them, λ 1 is an adjustable weight used to balance labeled training text samples and unlabeled training text samples. and are the categories of labeled training text samples and unlabeled training text samples predicted by the original classifier, respectively, lis the true category of the labeled training text samples.

[0145] Learn more consistent representations through instance-level contrastive learning, including:

[0146] Compute the contrastive loss for labeled training text samples via instance-level contrastive learning

[0147]

[0148] in, is a set of labeled training text samples, are all training text samples with the same label in the same batch; τ is a preset constant variable.

[0149] Computing the contrastive loss for unlabeled training text samples via instance-level contrastive learning

[0150]

[0151] in, is a set of unlabeled training text samples, Represents the enhanced training text sample, τ is a variable, referring to historical experience, τ = 0.07.

[0152] Contrasting loss l between labeled training text samples and unlabeled training text samples cont :

[0153]

[0154] Yes sup and l cont Perform weighted combination to get the final loss l:

[0155]

[0156] Among them, λ 2 yes The preset weights.

[0157] In a possible implementation, the method of using the Softmax function to classify the original output of the model and taking the category to which the Softmax maximum value belongs as the sample prediction category of the text sample includes: for the predicted text sample x, calculating the original output (Logits) L of the model:

[0158] L = g φ (f θ (x))

[0159] Among them, L is a K-dimensional vector, K represents the total number of categories; g φis the fully connected layer of the generalized intent discovery model, f θ BERT model for generalized intent discovery model.

[0160] Use the Softmax function to classify L, and predict the category of the text sample x to be the category to which the Softmax maximum value belongs:

[0161]

[0162] Among them, y is the predicted category of the predicted text sample x.

[0163] In a possible implementation, in order to measure the performance of the generalized intent discovery model, a comparative experiment was conducted on three widely used public datasets. The statistical characteristics of the datasets are shown in Table 1. k |、|Y n |、|D l |、|D u | and |D t |represent the number of known categories, new categories, labeled training text samples, unlabeled training text samples, and test data respectively.

[0164] Table 1

[0165]

[0166] The results of the comparative experiment are shown in Table 2:

[0167] Table 2

[0168]

[0169]

[0170] Table 2 is a comparison between the method proposed in the present invention and the existing methods on three different data sets, where Known represents the recognition accuracy of known categories, Novel represents the recognition accuracy of new categories, and H-score represents the harmonic mean of the accuracy of avoiding bias. From the data in Table 2, it can be seen that the method proposed in the present invention performs better than the existing methods on all data sets and indicators, and performs better than other existing methods in both the recognition of known categories and the discovery of new categories.

[0171] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0172] See also Figure 3In another embodiment of the present invention, a text generalized intent discovery system based on deviation self-correction calibration is provided, which can be used to implement the above-mentioned text generalized intent discovery method based on deviation self-correction calibration. Specifically, the text generalized intent discovery system based on deviation self-correction calibration includes a text sample model processing module and a text sample category prediction module.

[0173] Among them, the text sample model processing module is used to obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original output of the model; the text sample category prediction module is used to classify the original output of the model using the Softmax function, and take the category to which the Softmax maximum value belongs as the sample prediction category of the text sample; wherein, the generalized intent discovery model is obtained by the following method: constructing biased branches and trainable branches; wherein the biased branch includes a pre-trained BERT model and a pre-trained fully connected layer connected in sequence, and the trainable branch includes a BERT model and a fully connected layer connected in sequence; the parameters of the biased branch are fixed, and the parameters of the BERT model are initialized using the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized using the category prototype; for each training text sample, a biased output L is obtained through the biased branch bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, a migration matrix is ​​constructed and normalized to obtain a normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L that alleviates category confusion CCM ; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ]; According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And the true label Y of the labeled training text samples is calculated by the cross entropy loss function. l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l contA weighted combination is performed to obtain the final loss l; according to l, the parameters of the trainable branches are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

[0174] All relevant contents of each step involved in the aforementioned embodiment of the method for discovering generalized intent in text based on self-correction calibration of deviations can be referred to the functional description of the functional modules corresponding to the system for discovering generalized intent in text based on self-correction calibration of deviations in the embodiment of the present invention, and will not be repeated here.

[0175] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0176] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the text generalized intention discovery method based on deviation self-correction calibration.

[0177] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the text generalized intent discovery method based on deviation self-correction calibration in the above embodiment.

[0178] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for discovering generalized intent in text based on deviation self-correction calibration, characterized in that: include: Obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original output of the model; The Softmax function is used to classify the original output of the model, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample; Among them, the generalized intent discovery model is obtained in the following way: Construct biased branches and trainable branches; wherein the biased branches include sequentially connected pre-trained BERT models and pre-trained fully connected layers, and the trainable branches include sequentially connected BERT models and fully connected layers; The parameters of the biased branch are fixed, and the parameters of the BERT model are initialized with the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized with the category prototype; For each training text sample, a biased output L is obtained through a biased branch. bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, a migration matrix is ​​constructed and normalized to obtain a normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L that alleviates category confusion CCM ; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ]; According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And the true label Y of the labeled training text samples is calculated by the cross entropy loss function. l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont Perform weighted combination to obtain the final loss l; According to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

2. The method for discovering generalized intent of text based on deviation self-correction calibration according to claim 1 is characterized in that: The parameters of the fully connected layer initialized by using the category prototype include: For labeled training text samples, the category prototype is calculated by the following formula: in, represents the prototype of category j, x i is a training text sample; f θ is the BERT model; C j are all training text samples belonging to category j, and M is the number of labeled training text sample categories; For the unlabeled training text samples, KMeans clustering is first performed to obtain a set of all categories. K is the sum of the number of known categories and the number of unknown categories; For unlabeled training text samples, the category prototype is calculated by the following formula: in, Align the category prototypes of the unlabeled training text samples with the category prototypes of the labeled training text samples, and obtain the optimal arrangement of the category prototypes of the unlabeled training text samples by calculating the distance between the arrangement of the category prototypes of all unlabeled training text samples and the arrangement of the category prototypes of the labeled training text samples. in, Represents the calculation of the Euclidean distance between two vectors; Will Normalize and get the fully connected layer g φ Initial value of: g φ =[c1,...,c M ,...,c K ] Among them, the elements indexed from 1 to M represent the category prototypes of labeled training text samples.

3. The method for discovering generalized intent of text based on deviation self-correction calibration according to claim 1 is characterized in that: The biased output L is obtained by the biased branch bias , the training output L is obtained through the trainable branch θ , combined with a biased output weight factor α i Get the output L that alleviates the category bias CBM include: Calculate the training text sample x i Entropy E i : Among them, p ij =Softmax(L bias (x i )) is the training text sample x i Probability distribution on biased branches; According to the training text sample x i Entropy E i , the biased output weight factor α is obtained by the following formula = {α i }: a i =β·Sigmoid(E i -E max ) Among them, β is an adjustable weight, E max is the maximum entropy value in the current training text sample batch; The output L that alleviates the category bias is obtained by the following formula CBM : L CBM =L θ [:M]-α·L bias Among them, M represents the number of known categories.

4. The method for discovering generalized intent of text based on deviation self-correction calibration according to claim 1 is characterized in that: The method constructs a migration matrix according to the category prototype and normalizes it to obtain a normalized migration matrix, and outputs L according to the normalized migration matrix and the biased output bias and training output L θ Get the output L that alleviates category confusion CCM include: Construct the migration matrix T according to the category prototype M×N : Among them, {c1,...,c M } and {c M+1 ,...,c M+N } are prototypes of known and unknown categories; Normalize each row in the migration matrix to obtain the transition probability from each known category to the unknown category; The output L that alleviates category confusion is obtained by the following formula CCM : L CCM =L θ [M:]+α·L bias T ×Τ Among them, L θ [M:] represents L θ The last N elements of correspond to N unknown categories.

5. The method for discovering generalized text intent based on deviation self-correction calibration according to claim 1 is characterized in that: According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm include: The pseudo labels of unlabeled training text samples are obtained by the following formula Among them, Tr represents the rank of the matrix, ∈ is the weight factor, E represents the entropy function, B is the batch size, K is the number of categories, and vector 1 B and 1 K are unit vectors of dimension B and K respectively.

6. The method for discovering generalized text intent based on deviation self-correction calibration according to claim 1 is characterized in that: The cross entropy loss function is used to calculate the true label Y of the labeled training text samples. l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont The final loss l obtained by weighted combination includes: The true labels Y of the labeled training text samples are calculated by the cross entropy loss function l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup : Among them, λ1 is an adjustable weight, and are the categories of labeled training text samples and unlabeled training text samples predicted by the original classifier, respectively, l The true category of the labeled training text samples; Compute the contrastive loss for labeled training text samples via instance-level contrastive learning in, is a set of labeled training text samples, are all training text samples with the same label in the same batch; τ is a preset constant variable; Computing the contrastive loss for unlabeled training text samples via instance-level contrastive learning in, is a set of unlabeled training text samples, z i represents the enhanced training text samples; Contrasting loss l between labeled training text samples and unlabeled training text samples cont : Yes sup and l cont Perform weighted combination to get the final loss l: Where λ2 is The preset weights.

7. The method for discovering generalized intent of text based on deviation self-correction calibration according to claim 1 is characterized in that: The Softmax function is used to classify the original output of the model, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample, including: For the predicted text sample x, calculate the model's original output L: L=g φ (f θ (x)) Among them, L is a K-dimensional vector, K represents the total number of categories; g φ is the fully connected layer of the generalized intent discovery model, f θ BERT model for generalized intent discovery model; Use the Softmax function to classify L, and predict the category of the text sample x to be the category to which the Softmax maximum value belongs: Among them, y is the predicted category of the predicted text sample x.

8. A text generalized intent discovery system based on deviation self-correction calibration, characterized in that: include: The text sample model processing module is used to obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original output of the model; The text sample category prediction module is used to classify the original output of the model using the Softmax function, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample; Among them, the generalized intent discovery model is obtained in the following way: Construct biased branches and trainable branches; wherein the biased branches include sequentially connected pre-trained BERT models and pre-trained fully connected layers, and the trainable branches include sequentially connected BERT models and fully connected layers; The parameters of the biased branch are fixed, and the parameters of the BERT model are initialized with the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized with the category prototype; For each training text sample, a biased output L is obtained through a biased branch. bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, a migration matrix is ​​constructed and normalized to obtain a normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L that alleviates category confusion CCM ; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ]; According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And the true label Y of the labeled training text samples is calculated by the cross entropy loss function. l With predicted labels The loss between , and the pseudo labels of the unlabeled training text samples With predicted labels The loss between them is calculated by weighted combination to get the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont Perform weighted combination to obtain the final loss l; According to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for discovering generalized intent of text based on deviation self-correction calibration are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for discovering generalized intent of text based on deviation self-correction calibration are implemented as described in any one of claims 1 to 7.

Images