Method and related device for discovering generalized intent in text based on deviation self-correction calibration

Abstract

The present invention discloses a method and system for discovering generalized intent in text based on self-correction calibration of deviations, comprising: obtaining a predicted text sample and inputting it into a preset generalized intent discovery model to obtain the original output of the model; using the Softmax function to classify the original output of the model, and taking the category to which the Softmax maximum value belongs as the sample prediction category of the text sample. Among them, a biased branch and a trainable branch are set inside the generalized intent discovery model. First, a biased branch with a bias is obtained through pre-training, and the model parameters are fixed. Then, the training text sample is input into the pre-trained biased branch and the trainable branch respectively, and two original outputs are output. The original output of the biased branch is used to adjust the original output of the trainable branch, and the deviation of the model for known categories is used to alleviate category deviation and category confusion, which has a role in alleviating category deviation and category confusion, and effectively improves the recognition accuracy of the model for new categories.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing and relates to a method for discovering generalized intent in text based on deviation self-correction calibration and related devices. Background Art

[0002] Because many current deep learning models are trained on closed datasets, they can misidentify new intent categories. For example, in areas such as intent discovery and image classification, these trained models may encounter unlabeled training text samples that belong to categories they have never seen before.

[0003] To address these challenges in open scenarios, academics are currently proposing generalized intent discovery, which aims to identify known and unknown intent categories from unlabeled training text samples. The current mainstream approach to generalized intent discovery is to tailor or customize models pre-trained on labeled training text samples for specific tasks. However, in the absence of labeled data for new intent categories, these pre-trained models exhibit a strong bias toward known intent categories, leading to misclassification of samples from new intent categories as belonging to known intent categories.

[0004] To alleviate the bias towards known intent categories in existing methods, existing methods fine-tune the pre-trained model on unlabeled training text samples through pseudo-label training, contrastive learning, and transfer learning. Although the above methods have achieved certain improvements, they still have certain problems: (1) The model still has bias towards known intent categories, which may lead to the misclassification of new intent categories as known intent categories. (2) The model lacks effective supervision for new intent categories, which may lead to the misclassification of new intent categories as other new intent categories. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a text generalized intent discovery method and related devices based on deviation self-correction calibration.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for discovering generalized intent in text based on deviation self-correction calibration, comprising:

[0008] Obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the model's original output;

[0009] The Softmax function is used to classify the original output of the model, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample;

[0010] The generalized intent discovery model is obtained in the following way:

[0011] Construct biased branches and trainable branches; the biased branches include the pre-trained BERT model and the pre-trained fully connected layer connected in sequence, and the trainable branches include the BERT model and the fully connected layer connected in sequence;

[0012] The parameters of the biased branch are fixed, and the parameters of the BERT model are initialized with the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized with the category prototype;

[0013] For each training text sample, a biased output L is obtained through a biased branch bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, the migration matrix is ​​constructed and normalized to obtain the normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L to alleviate category confusion CCM ; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ];

[0014] According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And calculate the true label Y of the labeled training text sample respectively through the cross entropy loss function l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sip and l cont Perform weighted combination to obtain the final loss l;

[0015] According to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

[0016] Optionally, the parameters of the fully connected layer initialized using the category prototype include:

[0017] For labeled training text samples, the category prototype is calculated by the following formula:

[0018]

[0019] in, represents the prototype of category j, x i is the training text sample; f θ is the BERT model; C j are all training text samples belonging to category j, and M is the number of labeled training text sample categories;

[0020] For unlabeled training text samples, KMeans clustering is first performed to obtain a set of all categories K is the sum of the number of known categories and the number of unknown categories;

[0021] For unlabeled training text samples, the category prototype is calculated by the following formula:

[0022]

[0023] in,

[0024] Align the category prototypes of the unlabeled training text samples with the category prototypes of the labeled training text samples, and calculate the distance between the arrangement of the category prototypes of all unlabeled training text samples and the arrangement of the category prototypes of the labeled training text samples to obtain the optimal arrangement of the category prototypes of the unlabeled training text samples.

[0025]

[0026] in, Represents the calculation of the Euclidean distance between two vectors;

[0027] Will Normalize and get the fully connected layer g φ Initial value of:

[0028] g φ =[c1,...,c M ,...,c K ]

[0029] Among them, the elements indexed from 1 to M represent the category prototypes of the labeled training text samples.

[0030] Optionally, the biased output L is obtained by the biased branch bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α i Get the output L that alleviates the category deviation CBMinclude:

[0031] Calculate the training text sample x i Entropy E i :

[0032]

[0033] Among them, p ij =Softmax(L bias (x i )) is the training text sample x i Probability distribution on biased branches;

[0034] According to the training text sample x i Entropy E i , the biased output weight factor α is obtained by the following formula = {α i}:

[0035] α i =β·Sigmoid(E i -E max )

[0036] Among them, β is an adjustable weight, E max is the maximum entropy value in the current training text sample batch;

[0037] The output L that alleviates the category deviation is obtained by the following formula CBM :

[0038] L CBM =L θ [:M]-α·L bias

[0039] Where M represents the number of known categories.

[0040] Optionally, the migration matrix is ​​constructed according to the category prototype and normalized to obtain a normalized migration matrix, and the normalized migration matrix and the biased output L are obtained. bias and training output L θ Get the output L to alleviate category confusion CCM include:

[0041] Construct the migration matrix T according to the category prototype M×n :

[0042]

[0043] Among them, {c1,...,c M} and {c M+1 ,...,c M+N} are prototypes of known and unknown categories;

[0044] Normalize each row in the migration matrix to obtain the transition probability from each known category to the unknown category;

[0045] The output L that alleviates category confusion is obtained by the following formula CCM :

[0046] L CCM =L θ [M:]+α·L bias T ×Τ

[0047] Among them, L θ [M:] represents L θ The last N elements of correspond to N unknown categories.

[0048] Optionally, the L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm include:

[0049] The pseudo labels of unlabeled training text samples are obtained by the following formula

[0050]

[0051]

[0052] Among them, Tr represents the rank of the matrix, ∈ is the weight factor, E represents the entropy function, B is the batch size, K is the number of categories, and vector 1 B and 1 K are unit vectors of dimensions B and K respectively.

[0053] Optionally, the true labels Y of the labeled training text samples are calculated respectively by the cross entropy loss function. l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l uont The weighted combination to obtain the final loss l includes:

[0054] The true labels Y of the labeled training text samples are calculated separately through the cross entropy loss function l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup :

[0055]

[0056] Among them, λ1 is an adjustable weight, and are the categories of labeled training text samples and unlabeled training text samples predicted by the original classifier, Y l The true category of the labeled training text sample;

[0057] Compute the contrastive loss for labeled training text samples via instance-level contrastive learning

[0058]

[0059] in, is a set of labeled training text samples, are all training text samples with the same label in the same batch; τ is a preset constant variable;

[0060] Computing the contrastive loss for unlabeled training text samples via instance-level contrastive learning

[0061]

[0062] in, is a set of unlabeled training text samples, represents the enhanced training text samples;

[0063] The contrast loss l between labeled training text samples and unlabeled training text samples cont :

[0064]

[0065] Right sup and l cont Perform weighted combination to get the final loss l:

[0066]

[0067] Where λ2 is The preset weight of .

[0068] Optionally, the method of using the Softmax function to classify the original output of the model and taking the category to which the Softmax maximum value belongs as the sample prediction category of the text sample includes:

[0069] For the predicted text sample x, calculate the model's original output L:

[0070] L = g φ (f θ (x))

[0071] Among them, L is a K-dimensional vector, K represents the total number of categories; g φ is the fully connected layer of the generalized intent discovery model, f θ BERT model for generalized intent discovery model;

[0072] Use the Softmax function to classify L and predict the category of the text sample x to which the Softmax maximum value belongs:

[0073]

[0074] Among them, y is the predicted category of the predicted text sample x.

[0075] A second aspect of the present invention provides a system for discovering generalized intent in text based on deviation self-correction calibration, comprising:

[0076] The text sample model processing module is used to obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original model output;

[0077] The text sample category prediction module is used to classify the original output of the model using the Softmax function, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample;

[0078] The generalized intent discovery model is obtained in the following way:

[0079] Construct biased branches and trainable branches; the biased branches include the pre-trained BERT model and the pre-trained fully connected layer connected in sequence, and the trainable branches include the BERT model and the fully connected layer connected in sequence;

[0080] The parameters of the biased branch are fixed, and the parameters of the BERT model are initialized with the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized with the category prototype;

[0081] For each training text sample, a biased output L is obtained through a biased branch bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, the migration matrix is ​​constructed and normalized to obtain the normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L to alleviate category confusion CCM; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ];

[0082] According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And calculate the true label Y of the labeled training text sample respectively through the cross entropy loss function l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont Perform weighted combination to obtain the final loss l;

[0083] According to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

[0084] In a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned method for discovering generalized intent in text based on self-correction calibration of deviations are implemented.

[0085] In a fourth aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned method for discovering generalized intent of text based on deviation self-correction calibration are implemented.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] The present invention is a method for discovering generalized intent in text based on bias self-correction calibration. The method obtains a predicted text sample and inputs it into a preset generalized intent discovery model to obtain the model's original output. The model's original output is then classified using the Softmax function, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample, thereby effectively identifying the new intent category of the predicted text sample. The generalized intent discovery model is internally provided with a biased branch and a trainable branch. First, a biased biased branch is obtained through pre-training, and the model parameters are fixed. The training text sample is then input into the pre-trained biased branch and the trainable branch, respectively, and two original outputs are output. The original output of the biased branch is used to adjust the original output of the trainable branch, and the model's deviation from known categories is used to mitigate category bias and category confusion. Furthermore, the present invention proposes a method for adjusting the output of the biased model, which mitigates both category bias and category confusion, effectively improving the model's recognition accuracy for new categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 This is a flow chart of a method for discovering generalized intent in text based on deviation self-correction calibration according to an embodiment of the present invention.

[0089] Figure 2 Schematic diagram of the training principle of the generalized intent discovery model according to an embodiment of the present invention.

[0090] Figure 3 This is a structural block diagram of a text generalized intent discovery system based on deviation self-correction calibration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0092] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0093] The present invention is described in further detail below with reference to the accompanying drawings:

[0094] See also Figure 1 In one embodiment of the present invention, a method for discovering generalized text intent based on deviation self-correction calibration is provided to improve the recognition accuracy of new categories of text samples.

[0095] Specifically, the method for discovering generalized text intent based on deviation self-correction calibration of the present invention includes the following steps:

[0096] S1: Obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original output of the model.

[0097] S2: Use the Softmax function to classify the original output of the model, and use the category to which the Softmax maximum value belongs as the sample prediction category of the text sample.

[0098] The generalized intent discovery model is obtained by constructing a biased branch and a trainable branch; wherein the biased branch includes a pre-trained BERT model and a pre-trained fully connected layer connected in sequence, and the trainable branch includes a BERT model and a fully connected layer connected in sequence; the parameters of the biased branch are fixed, and the parameters of the BERT model are initialized using the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized using the category prototype; for each training text sample, a biased output L is obtained through the biased branch. bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, the migration matrix is ​​constructed and normalized to obtain the normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L to alleviate category confusion CCM ; Merge L CBM and L CCMGet the deviation self-calibration output L c =[L CBM ,L CCM ]; According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And calculate the true label Y of the labeled training text sample respectively through the cross entropy loss function l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont A weighted combination is performed to obtain the final loss l; according to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

[0099] The present invention is a method for discovering generalized intent in text based on bias self-correction calibration. The method obtains a predicted text sample and inputs it into a preset generalized intent discovery model to obtain the model's original output. The model's original output is then classified using the Softmax function, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample, thereby effectively identifying the new intent category of the predicted text sample. The generalized intent discovery model is internally provided with a biased branch and a trainable branch. First, a biased biased branch is obtained through pre-training, and the model parameters are fixed. The training text sample is then input into the pre-trained biased branch and the trainable branch, respectively, and two original outputs are output. The original output of the biased branch is used to adjust the original output of the trainable branch, and the model's deviation from known categories is used to mitigate category bias and category confusion. Furthermore, the present invention proposes a method for adjusting the output of the biased model, which mitigates both category bias and category confusion, effectively improving the model's recognition accuracy for new categories.

[0100] In one possible implementation, training the generalized intent discovery model includes the following detailed steps:

[0101] 1. Obtain biased model f through pre-training bias and biased classifier g bias , where the biased model f biasTo pre-train the BERT model, the biased classifier is a pre-trained fully connected layer. During pre-training, the cross entropy loss function is used for labeled training text samples, and the MLM (masked language modeling) loss function is used for unlabeled training text samples. The biased model f bias : and biased classifier g bias : Among them, d is the feature dimension and M is the number of known categories.

[0102] 2. Freeze the pre-trained biased model f bias and biased classifier g bias Parameters. Using a trainable model f θ With a trainable classifier g φ Used together for training the entire model. θ is with f bias BERT model with the same structure, g φ is a fully connected layer, θ and φ are learnable parameters. bias Parameters to initialize f θ , use the category prototype to initialize the classifier g φ .

[0103] The method for calculating the category prototype is as follows: the KMeans clustering algorithm is used on the unlabeled training text samples, and each category center obtained by KMeans is regarded as the category prototype. Then, the alignment algorithm is used to align the known categories in g bias and g φ Then, g φ Initialized to the aligned category prototype after l2 normalization: g φ =[c1,...,c M ,...,c K ], where indexes 1 to M represent known category prototypes.

[0104] 3. For each training text sample x i , first input into the pre-trained biased model f bias and biased classifier g bias , and get the biased output L bias ; Then input to the trainable model f θ and a trainable classifier g φ , we get the output L θ Calculate the biased output weight factor α i . Use L θ Subtract α·L bias , and obtain the output L that reduces the model category deviation CBM .

[0105] 4. Use the category prototype to measure the similarity between categories and calculate the migration matrix T based on the category prototype M×N , M and N represent the number of known categories and unknown categories respectively. Then the migration matrix T M×N Normalize to obtain the transition probability from each known category to the new category. According to the migration matrix T, the output L after mitigating the category confusion is obtained. CCM Combine the output of mitigating category deviation and the output of mitigating category confusion to obtain the final bias self-calibration output L c =[L CBM ,L CCM ].

[0106] 5. According to the calibrated output L c , the pseudo labels of the unlabeled training text samples are obtained by the SK algorithm. The true labels Y of the labeled training text samples are calculated by the cross entropy loss function. l and predicted labels The loss between the pseudo labels of unlabeled training text samples and predicted labels The loss between them is weighted and combined to get the total cross entropy loss l sup . Use instance-level contrastive learning to calculate the contrast loss l between labeled training text samples and unlabeled training text samples cont . sup and l cont Perform weighted combination to get the final loss value l. Update the model parameters continuously through back propagation until l converges to get the trained model f θ and classifier g φ , that is, the generalized intent discovery model.

[0107] In a possible implementation, the method of initializing the parameters of the fully connected layer using the category prototype includes:

[0108] For labeled training text samples, the category prototype is calculated by the following formula:

[0109]

[0110] in, represents the prototype of category j, x i is the training text sample; f θ is the BERT model; C j are all training text samples belonging to category j, and M is the number of labeled training text sample categories.

[0111] For unlabeled training text samples, KMeans clustering is first performed to obtain a set of all categories K is the sum of the number of known categories and the number of unknown categories.

[0112] For unlabeled training text samples, the category prototype is calculated by the following formula:

[0113]

[0114] in,

[0115] Align the category prototypes of the unlabeled training text samples with the category prototypes of the labeled training text samples, and calculate the distance between the arrangement of the category prototypes of all unlabeled training text samples and the arrangement of the category prototypes of the labeled training text samples to obtain the optimal arrangement of the category prototypes of the unlabeled training text samples.

[0116]

[0117] in, Represents the calculation of the Euclidean distance between two vectors.

[0118] Will Normalize and get the fully connected layer g φ Initial value of:

[0119] g φ =[c1,...,c M ,...,c K ]

[0120] Among them, the elements indexed from 1 to M represent the category prototypes of the labeled training text samples.

[0121] In a possible implementation manner, the biased output L is obtained by the biased branch. bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α i Get the output L that alleviates the category deviation CBM Includes: Calculate training text sample x i Entropy E i :

[0122]

[0123] Among them, p ij =Softmax(L bias (x i )) is the training text sample x i Probability distribution over biased branches.

[0124] According to the training text sample x i Entropy E i, the biased output weight factor α is obtained by the following formula = {α i}:

[0125] α i =β·Sigmoid(E i -E max )

[0126] Among them, β is an adjustable weight, E max is the maximum entropy value in the current training text sample batch.

[0127] The output L that alleviates the category deviation is obtained by the following formula CBM :

[0128] L CBM =L θ [:M]-α·L bias

[0129] Where M represents the number of known categories.

[0130] In a possible implementation, the process of constructing a migration matrix according to the category prototype and normalizing the matrix to obtain a normalized migration matrix, and the process of generating a biased output L according to the normalized migration matrix and the biased output L bias and training output L θ Get the output L to alleviate category confusion CCM Including: Constructing the migration matrix T according to the category prototype M×N :

[0131]

[0132] Among them, {c1,...,c M} and {c M+1 ,...,c M+N} are prototypes of known and unknown categories;

[0133] Normalize each row in the migration matrix to obtain the transition probability from each known category to the unknown category;

[0134] The output L that alleviates category confusion is obtained by the following formula CCM :

[0135] L CCM =L θ [M:]+α·L bias T ×T

[0136] Among them, L θ [M:] represents L θ The last N elements of correspond to N unknown categories.

[0137] In a possible implementation manner, the c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm Including: Get the pseudo label of the unlabeled training text sample by the following formula

[0138]

[0139]

[0140] Among them, Tr represents the rank of the matrix, ∈ is the weight factor, E represents the entropy function, B is the batch size, K is the number of categories, and vector 1 B and 1 K are unit vectors of dimensions B and K respectively.

[0141] In one possible implementation, the cross entropy loss function is used to calculate the true labels Y of the labeled training text samples. l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples xont ; for l sup and l cont The weighted combination to obtain the final loss l includes:

[0142] The true labels Y of the labeled training text samples are calculated separately through the cross entropy loss function l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup :

[0143]

[0144] Among them, λ1 is an adjustable weight used to balance the labeled training text samples and the unlabeled training text samples. and are the categories of labeled training text samples and unlabeled training text samples predicted by the original classifier, Y l is the true category of the labeled training text samples.

[0145] Learn more consistent representations through instance-level contrastive learning, specifically including:

[0146] Compute the contrastive loss for labeled training text samples via instance-level contrastive learning

[0147]

[0148] in, is a set of labeled training text samples, are all training text samples with the same label in the same batch; τ is a preset constant variable.

[0149] Computing the contrastive loss for unlabeled training text samples via instance-level contrastive learning

[0150]

[0151] in, is a set of unlabeled training text samples, Represents the enhanced training text sample, τ is a variable, referring to historical experience, τ = 0.07.

[0152] The contrast loss l between labeled training text samples and unlabeled training text samples cont :

[0153]

[0154] Right sup and l cont Perform weighted combination to get the final loss l:

[0155]

[0156] Where λ2 is The preset weight of .

[0157] In a possible implementation, the method of using the Softmax function to classify the original output of the model and taking the category to which the Softmax maximum value belongs as the sample prediction category of the text sample includes: for the predicted text sample x, calculating the original output (Logits) L of the model:

[0158] L = g φ (f θ (x))

[0159] Among them, L is a K-dimensional vector, K represents the total number of categories; g φ is the fully connected layer of the generalized intent discovery model, f θ BERT model for generalized intent discovery.

[0160] Use the Softmax function to classify L and predict the category of the text sample x to which the Softmax maximum value belongs:

[0161]

[0162] Among them, y is the predicted category of the predicted text sample x.

[0163] In a possible implementation, in order to measure the performance of the generalized intent discovery model, a comparative experiment was conducted on three widely used public datasets. The statistical characteristics of the datasets are shown in Table 1. k |、|Y n |、|D l |、|D u | and |D t |represent the number of known categories, new categories, labeled training text samples, unlabeled training text samples, and test data respectively.

[0164] Table 1

[0165]

[0166] The results of the comparative experiment are shown in Table 2:

[0167] Table 2

[0168]

[0169]

[0170] Table 2 compares the proposed method with existing methods on three different datasets. Known represents the recognition accuracy for known classes, Novel represents the recognition accuracy for novel classes, and H-score represents the harmonic mean of the accuracy rates for avoiding bias. The data in Table 2 show that the proposed method outperforms existing methods across all datasets and metrics, outperforming existing methods in both recognizing known classes and discovering novel classes.

[0171] The following are device embodiments of the present invention, which can be used to perform the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0172] See also Figure 3 In another embodiment of the present invention, a text generalized intent discovery system based on deviation self-correction calibration is provided, which can be used to implement the above-mentioned text generalized intent discovery method based on deviation self-correction calibration. Specifically, the text generalized intent discovery system based on deviation self-correction calibration includes a text sample model processing module and a text sample category prediction module.

[0173] Among them, the text sample model processing module is used to obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original output of the model; the text sample category prediction module is used to classify the original output of the model using the Softmax function, and take the category to which the Softmax maximum value belongs as the sample prediction category of the text sample; wherein, the generalized intent discovery model is obtained by the following method: constructing a biased branch and a trainable branch; wherein, the biased branch includes a pre-trained BERT model and a pre-trained fully connected layer connected in sequence, and the trainable branch includes a BERT model and a fully connected layer connected in sequence; fixing the parameters of the biased branch, and using the parameters of the pre-trained BERT model to initialize the parameters of the BERT model, and using the category prototype to initialize the parameters of the fully connected layer; for each training text sample, a biased output L is obtained through the biased branch. bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, the migration matrix is ​​constructed and normalized to obtain the normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L to alleviate category confusion CCM ; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ]; According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And calculate the true label Y of the labeled training text sample respectively through the cross entropy loss function l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont A weighted combination is performed to obtain the final loss l; according to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

[0174] All relevant contents of each step involved in the embodiment of the aforementioned method for discovering generalized intentions in text based on self-correction calibration of deviations can be referred to the functional description of the functional modules corresponding to the system for discovering generalized intentions in text based on self-correction calibration of deviations in the embodiment of the present invention, and will not be repeated here.

[0175] The module division in the embodiments of the present invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in various embodiments of the present invention may be integrated into a single processor, exist physically as separate modules, or two or more modules may be integrated into a single module. The integrated modules may be implemented in either hardware or software functional modules.

[0176] In another embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the text generalized intent discovery method based on deviation self-correction calibration.

[0177] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the text generalized intent discovery method based on deviation self-correction calibration in the above embodiment.

[0178] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for discovering generalized intent in text based on deviation self-correction calibration, characterized in that: include: Obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the model's original output; The Softmax function is used to classify the original output of the model, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample; The generalized intent discovery model is obtained in the following way: Construct biased branches and trainable branches; the biased branches include the pre-trained BERT model and the pre-trained fully connected layer connected in sequence, and the trainable branches include the BERT model and the fully connected layer connected in sequence; The parameters of the biased branch are fixed, and the parameters of the BERT model are initialized with the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized with the category prototype; For each training text sample, a biased output L is obtained through a biased branch bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, the migration matrix is ​​constructed and normalized to obtain the normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L to alleviate category confusion CCM ; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ]; According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And calculate the true label Y of the labeled training text sample respectively through the cross entropy loss function l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont Perform weighted combination to obtain the final loss l; According to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

2. The method for discovering generalized text intent based on deviation self-correction calibration according to claim 1 is characterized in that: The parameters of the fully connected layer initialized using category prototypes include: For labeled training text samples, the category prototype is calculated by the following formula: in, represents the prototype of category j, x i is the training text sample; f θ is the BERT model; C j are all training text samples belonging to category j, and M is the number of labeled training text sample categories; For unlabeled training text samples, KMeans clustering is first performed to obtain a set of all categories K is the sum of the number of known categories and the number of unknown categories; For unlabeled training text samples, the category prototype is calculated by the following formula: in, Align the category prototypes of the unlabeled training text samples with the category prototypes of the labeled training text samples, and calculate the distance between the arrangement of the category prototypes of all unlabeled training text samples and the arrangement of the category prototypes of the labeled training text samples to obtain the optimal arrangement of the category prototypes of the unlabeled training text samples. in, Represents the calculation of the Euclidean distance between two vectors; Will Normalize and get the fully connected layer g φ Initial value of: g φ =[c1,...,c M ,...,c K ] Among them, the elements indexed from 1 to M represent the category prototypes of the labeled training text samples.

3. The method for discovering generalized text intent based on deviation self-correction calibration according to claim 1 is characterized in that: The biased output L is obtained by the biased branch. bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α i Get the output L that alleviates the category deviation CBM include: Calculate the training text sample x i Entropy E i : Among them, p ij =Softmax(L bias (x i )) is the training text sample x i Probability distribution on biased branches; According to the training text sample x i Entropy E i , the biased output weight factor α is obtained by the following formula = {α i }: a i =β·Sigmoid(E i -E max ) Among them, β is an adjustable weight, E max is the maximum entropy value in the current training text sample batch; The output L that alleviates the category deviation is obtained by the following formula CBM : L CBM =L θ [:M]-α·L bias Where M represents the number of known categories.

4. The method for discovering generalized text intent based on deviation self-correction calibration according to claim 1 is characterized in that: The transfer matrix is ​​constructed according to the category prototype and normalized to obtain the normalized transfer matrix, and the biased output L is obtained according to the normalized transfer matrix. bias and training output L θ Get the output L to alleviate category confusion CCM include: Construct the migration matrix T according to the category prototype M×N : Among them, {c1,...,c M } and {c M+1 ,...,c M+N } are prototypes of known and unknown categories; Normalize each row in the migration matrix to obtain the transition probability from each known category to the unknown category; The output L that alleviates category confusion is obtained by the following formula CCM : L CCM =L θ [M:]+α·L bias T ×Τ Among them, L θ [M:] represents L θ The last N elements of correspond to N unknown categories.

5. The method for discovering generalized text intent based on deviation self-correction calibration according to claim 1 is characterized in that: According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm include: The pseudo labels of unlabeled training text samples are obtained by the following formula Among them, Tr represents the rank of the matrix, ∈ is the weight factor, E represents the entropy function, B is the batch size, K is the number of categories, and vector 1 B and 1 K are unit vectors of dimensions B and K respectively.

6. The method for discovering generalized text intent based on deviation self-correction calibration according to claim 1 is characterized in that: The cross entropy loss function is used to calculate the true label Y of the labeled training text sample. l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont The weighted combination to obtain the final loss l includes: The true labels Y of the labeled training text samples are calculated separately through the cross entropy loss function l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup : Among them, λ1 is an adjustable weight, and are the categories of labeled training text samples and unlabeled training text samples predicted by the original classifier, Y l The true category of the labeled training text sample; Compute the contrastive loss for labeled training text samples via instance-level contrastive learning in, is a set of labeled training text samples, are all training text samples with the same label in the same batch; τ is a preset constant variable; Computing the contrastive loss for unlabeled training text samples via instance-level contrastive learning in, is a set of unlabeled training text samples, z i represents the enhanced training text samples; The contrast loss l between labeled training text samples and unlabeled training text samples cont : Right sup and l cont Perform weighted combination to get the final loss l: Where λ2 is The preset weight of .

7. The method for discovering generalized text intent based on deviation self-correction calibration according to claim 1 is characterized in that: The Softmax function is used to classify the original output of the model, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample, including: For the predicted text sample x, calculate the model's original output L: L=g φ (f θ (x)) Among them, L is a K-dimensional vector, K represents the total number of categories; g φ is the fully connected layer of the generalized intent discovery model, f θ BERT model for generalized intent discovery model; Use the Softmax function to classify L and predict the category of the text sample x to which the Softmax maximum value belongs: Among them, y is the predicted category of the predicted text sample x.

8. A text generalized intent discovery system based on deviation self-correction calibration, characterized by: include: The text sample model processing module is used to obtain the predicted text sample and input it into the preset generalized intent discovery model to obtain the original model output; The text sample category prediction module is used to classify the original output of the model using the Softmax function, and the category to which the Softmax maximum value belongs is used as the sample prediction category of the text sample; The generalized intent discovery model is obtained in the following way: Construct biased branches and trainable branches; the biased branches include the pre-trained BERT model and the pre-trained fully connected layer connected in sequence, and the trainable branches include the BERT model and the fully connected layer connected in sequence; The parameters of the biased branch are fixed, and the parameters of the BERT model are initialized with the parameters of the pre-trained BERT model, and the parameters of the fully connected layer are initialized with the category prototype; For each training text sample, a biased output L is obtained through a biased branch bias , the training output L is obtained through the trainable branch θ , and combined with the biased output weight factor α to obtain the output L that alleviates the category bias CBM ; According to the category prototype, the migration matrix is ​​constructed and normalized to obtain the normalized migration matrix, and according to the normalized migration matrix, the biased output L bias and training output L θ Get the output L to alleviate category confusion CCM ; Merge L CBM and L CCM Get the deviation self-calibration output L c =[L CBM ,L CCM ]; According to L c The pseudo labels of unlabeled training text samples are obtained through the SK algorithm And calculate the true label Y of the labeled training text sample respectively through the cross entropy loss function l and predicted labels The loss between , and the pseudo labels of the unlabeled training text samples and predicted labels The loss between them is calculated and the weighted combination of the two is used to obtain the cross entropy loss l sup ; and use instance-level contrastive learning to calculate the contrastive loss l between labeled training text samples and unlabeled training text samples cont ; for l sup and l cont Perform weighted combination to obtain the final loss l; According to l, the parameters of the trainable branch are continuously updated through back propagation until l converges, and the final trainable branch is used as the preset generalized intent discovery model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for discovering generalized intent of text based on deviation self-correction calibration are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for discovering generalized intent of text based on deviation self-correction calibration are implemented as described in any one of claims 1 to 7.

Images