Small sample user multi-intention recognition method with reinforced correlation degree calculation

Abstract

The present application relates to a kind of small sample user multi-intention recognition method of reinforcing correlation degree calculation, belong to natural language processing field.The present application is intended to label multiple intention labels for user dialogue sample, first, the embedding representation of user dialogue sample and its corresponding label is extracted using BERT;Second, the correlation degree of label and sample in support set is calculated to enhance label prototype representation, and the prototype network is constructed using label prototype and query sample;Then, the universal threshold of multiple fields and the specific threshold of small sample field are calculated, and the correlation degree average of other field samples and query sample is used as coefficient to calculate the threshold of prototype network;Finally, multiple intention labels corresponding to query sample are filtered according to the threshold of prototype network.The present application proposes to strengthen the correlation degree calculation method, considering the feature value difference and feature dimension difference of embedding representation, further reduces the interference between labels, and improves the threshold calculation method of prototype network, effectively improves the accuracy of multi-intention recognition task.

Description

Technical Field

[0001] This invention relates to a method for identifying multiple intents of users in small samples with enhanced correlation calculation, and particularly to the technical field of small sample multi-label learning, belonging to the field of natural language processing. Background Technology

[0002] Intent detection, a key task in human-computer interaction (HCI) applications, aims to determine a user's true intent from their dialogue. Since user dialogue contains rich semantic information, the content expressed by a user may often contain multiple intents. However, the cost of collecting and classifying real user dialogue is high, resulting in a small number of labeled samples and a tendency for classification models to overfit. Furthermore, the semantic structure of text is complex, and a single dialogue may contain multiple intents simultaneously. Therefore, this paper extends the concept of multi-label few-shot learning to the field of intent detection, training a multi-intent recognition model with a small number of labeled samples. Intent types are represented as labels, and the multi-intent recognition model is used to assign multiple relevant labels to user dialogues, enabling the HCI model to better understand user intent.

[0003] The goal of few-shot multi-label learning is to mimic human cognitive learning abilities, enabling classification models to quickly learn new sample features with only a few labeled samples, and assign the most relevant subset of labels to a given sample. Based on whether or not they rely on additional knowledge, recent few-shot multi-label learning methods can be broadly categorized into methods that rely on additional knowledge and methods based on prototype networks.

[0004] 1. Methods relying on additional knowledge

[0005] Methods that rely on additional knowledge encode labels using predefined information, such as the hierarchical structure or descriptive information of the labels, to help build classification models. However, these methods depend on a structured label space and label description information, and the predefined information requires domain-specific expertise, thus limiting their application scope.

[0006] 2. Prototype Network-Based Methods

[0007] The prototype network-based method constructs data into a meta-task form, obtains the prototype representation of the label through the support set sample representation, and measures the distance or similarity between the query sample and each label prototype to match the label related to the sample. However, the small-sample multi-label classification method using prototype networks has two problems: (1) Noise interference. The embedded representation of the sample contains noise information introduced by the corresponding multi-label. When only the mean of the support set sample representation is used as the label prototype representation, the difference between the label prototype representations will be weakened, leading to confusion between multiple label prototypes. When measuring the distance between the query sample representation and the label prototype representation, the noise information contained in the query sample will affect the accuracy of the distance measurement. (2) Threshold bias. Existing methods using fixed thresholds are difficult to adapt to diverse user dialogue domains, which may result in the threshold being unsuitable for small-sample domains. On the other hand, the method of calculating the threshold using kernel regression does not make full use of prior knowledge, which may result in the threshold only satisfying a few samples.

[0008] In summary, to address the limitations of existing methods that rely on additional knowledge and the problems of noise interference and biased calculation thresholds in prototype network-based methods, this invention proposes a small-sample user multi-intent recognition method that improves prototype network-based enhanced correlation calculation. Summary of the Invention

[0009] The purpose of this invention is to meet the practical needs of user multi-intent recognition when labeled samples are insufficient, improve the problem of limited application scope of methods that rely on additional knowledge, and overcome the problems of noise information introduced by multiple labels in the sample representation when training a small sample user multi-intent recognition model using a prototype network and insufficient utilization of prior knowledge when calculating the threshold.

[0010] The design principle of this invention is as follows: First, the pre-trained model BERT is used to extract the user dialogue sample set and its corresponding label embedding representation, and the samples are reconstructed into a meta-task form of few-shot learning (N-way K-shot); Second, a method for strengthening the correlation calculation is proposed to calculate the label prototype representation under few-shot conditions, feature-level attention is introduced to calculate the label prototype radius, and a prototype network is constructed; Then, the method for strengthening the correlation calculation is used to generate a general threshold under multiple domains and a specific threshold under few-shot conditions, and the threshold of the prototype network is calculated; Finally, multiple intent labels corresponding to the query samples are selected based on the threshold of the prototype network.

[0011] The technical solution of the present invention is achieved through the following steps:

[0012] Step 1: Obtain the user dialogue sample set and its corresponding label embedding representation, and reconstruct the user dialogue sample set into a user multi-intent recognition task scenario under few-shot learning.

[0013] Step 2: Calculate the prototype representation and prototype radius of the label under small sample conditions, and construct the prototype network using the query sample representation, label prototype representation, and label prototype radius.

[0014] Step 3: Use user dialogue samples with labeled information from other domains that are different from the small sample domain as prior knowledge to calculate the general threshold.

[0015] Step 4: Calculate a specific threshold based on the correlation between user dialogue samples in this small sample domain, and use the specific threshold to calibrate the general threshold to calculate the threshold of the prototype network.

[0016] Step 5: Filter out multiple intent labels corresponding to the query samples based on the threshold of the prototype network.

[0017] Beneficial effects

[0018] Compared to methods that rely on additional knowledge, this invention does not rely on predefined information as additional knowledge. Instead, it learns prototype representations and measures the distance between query samples and prototypes from a specific domain. It combines samples from multiple domains and a specific domain to obtain a threshold for that specific domain, thus expanding the scope of application and better reflecting the diverse realities of dialogue sample domains.

[0019] Compared to prototype network-based methods, this invention improves the prototype generation and threshold calculation methods, enhancing the accuracy of multi-intent recognition for users with small sample sizes. Regarding prototype generation, this invention proposes an enhanced correlation calculation method to reduce noise interference introduced by other labels in the sample representation and increase the difference between label prototype representations. In threshold calculation, it fully utilizes the enhanced correlation calculation method to strengthen the use of prior knowledge, making the general threshold more adaptive. Furthermore, it generates specific thresholds based on the characteristics of small-sample domain samples, using these specific thresholds to correct the general threshold and calculate the threshold for the prototype network, thus improving the accuracy of threshold calculation. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the principle framework of the small-sample user multi-intent recognition method for enhancing correlation calculation in this invention.

[0021] Figure 2 Flowchart for data reconstruction of samples and labels.

[0022] Figure 3 An example of data reconstruction for samples and labels.

[0023] Figure 4 A schematic diagram illustrating the calculation of weight coefficients using feature-level attention.

[0024] Figure 5 This is a schematic diagram of the prototype network. Detailed Implementation

[0025] To better illustrate the purpose and advantages of the present invention, the embodiments of the method of the present invention will be described in further detail below.

[0026] The specific process is as follows:

[0027] Step 1: Obtain the user dialogue sample set and its corresponding label embedding representation. Reconstruct the user dialogue sample set into a meta-task form (N-way K-shot) for user multi-intent recognition under small sample size. Each meta-task contains N labels, each label corresponds to K samples, and the support set samples are composed of N×K corresponding samples.

[0028] Step 1.1: Use the pre-trained model BERT to extract the embedding representation x of user dialogue samples in each domain and the embedding representation y of their corresponding labels.

[0029] Step 1.2, from the label set corresponding to the sample set Θ of a certain domain. Extract N tags from the set to form a tag subset L.

[0030] Step 1.3: Extract one tag l from the tag subset L without replacement.

[0031] Step 1.4: Select a subset Θ′ of samples corresponding to label l from the sample set Θ. Multiple labels corresponding to each sample in the sample subset Θ′ are extracted, and multiple sample-multi-label pairs are formed according to the correspondence between the sample subset Θ′ and the labels.

[0032] Step 1.5: Extract one label without replacement from the sample-multi-label pair in 1.4 and add it to the candidate set P. Calculate the number of non-repeating labels n in P.

[0033] Step 1.6: If the number of tags n is less than K, then proceed to step 1.5; otherwise, proceed to step 1.7.

[0034] Step 1.7: If the tag subset L is not empty, then execute step 1.3; otherwise, execute step 1.8.

[0035] Step 1.8: Extract one sample-multi-label pair from the candidate set P without replacement, and recalculate the number of non-repeating labels n.

[0036] Step 1.9: If n is less than K, then put the sample with multiple labels into the support set S, put the corresponding multiple labels into the label subset Y of the support set, and delete the duplicate labels in the label subset Y.

[0037] Step 1.10: Execute step 1.8 until the candidate set P is empty. The process from steps 1.2 to 1.10 is as follows: Figure 2 As shown, an example of this process is illustrated as follows: Figure 3 As shown.

[0038] Step 1.11: Copy the samples that are shared by multiple labels in the support set S until the support set S is expanded into the standard K-shot form.

[0039] Step 1.12: Randomly select one label from the label subset Y, and from the sample set... One sample corresponding to the label is randomly selected from the set as the query sample. It is the complement of the support set S.

[0040] Step 2: Calculate the prototype representation and prototype radius of the labels corresponding to the support set samples, and construct the prototype network using the query sample representation, label prototype representation, and label prototype radius.

[0041] Step 2.1: Define the reinforcement association calculation function f(u, v) to solve for the association score between the embedding representations u and v, using the cosine distance as the coefficient of the Euclidean distance, while considering the differences in feature values ​​and feature dimensions between the embedding representations. The definition is given in formula (1). Calculate the embedding representations of the K support set samples corresponding to label i in the label subset Y. The correlation score between the embedding representation yi∈y (i∈[1,N]) of label i and the label i is related to the yi∈y (i∈[1,N]). The calculation formula is shown in formula (2).

[0042]

[0043]

[0044] Step 2.2, for the embedding representation of support set samples Assign corresponding relevance weights Strengthen the focus on support set samples that are highly correlated with label i. The calculation formula is shown in formula (3).

[0045]

[0046] Step 2.3, embedding representation of the support set samples corresponding to label i. Perform a weighted summation and proportionally mix the embedding representations y of label i. i Embedded representation of support set samples Obtain the prototype representation c of label i i The calculation formula is shown in formula (4).

[0047]

[0048] Among them, the mean of the relevance score The weighted sum of the embedding representations of the support set samples is used to generate the label prototype c.i The weight of time is calculated using formula (5).

[0049]

[0050] Step 2.4 introduces a feature-level attention mechanism to calculate the similarity between the support set samples corresponding to label i, denoted as the weight coefficient z. i The specific process is as follows: using the embedding representation of the support set sample corresponding to label i. As input, the embedding representations of the support set samples are convolved three times using a K×1 convolution kernel, with strides of 1×1, 1×1, and K×1, respectively. After each convolution, a ReLU activation function is applied to obtain the weight coefficients z. i The specific calculation process is as follows: Figure 4 As shown.

[0051] Step 2.5, using the weighting coefficient z i The prototype representation of the highlighted label i is c. i The more discriminative feature dimension is calculated by interpolating the embedding representation x0 of the query sample and the tag prototype representation c. i The distance between them is used as the prototype radius s of label i. i The prototype radius of the label is s i The calculation formula is shown in formula (6).

[0052] s i =-(z i ·(x0-c i ) 2 ), i∈[1,N] (6)

[0053] Step 2.6: Using the query sample representation x0 as the center of the prototype network, the label prototype radius s i As a query sample, x0 represents the label prototype representation c. i The distance between them is represented by the label prototype c. i Determine the specific location of the label prototype to construct the prototype network.

[0054] Step 3: Use user dialogue samples with labeled information from other domains that are different from the small sample domain as prior knowledge to calculate the general threshold.

[0055] Step 3.1: Extract R user dialogue samples from other domains different from the small sample domain as prior knowledge, and calculate the embedding representation of sample r. Embedded expressions of each label in the domain to which sample r belongs correlation score between and correlation weight To give different fields of threshold adaptability, the calculation formulas are shown in formula (7) and formula (8).

[0056]

[0057]

[0058] in, The score represents the correlation between the current sample r and the label l, where M is the number of labels in this domain.

[0059] Step 3.2, calculate the variance of the association weights between sample r and each label. With variance The difficulty of distinguishing the label corresponding to sample r in the neighborhood of sample r is measured by formula (9).

[0060]

[0061] Step 3.3: Based on prior knowledge, the label corresponding to sample r can be determined, and the relevance weight of the corresponding label is denoted as... Based on the lowest relevance weight among the corresponding tags and variance Calculate the general threshold t meta The calculation formula is shown in formula (10).

[0062]

[0063] Step 4: Calculate a specific threshold based on the correlation between user dialogue samples in a small sample domain, and use the specific threshold to calibrate the general threshold to calculate the threshold of the prototype network.

[0064] Step 4.1, based on the label prototype radius s i The sizes are sorted from smallest to largest according to the label prototype.

[0065] Step 4.2: Filter the multiple samples corresponding to each label i∈[1,N] in the support set S in the order of label prototypes, and represent them with the embeddings of the corresponding samples. The embedding representation x0 of the query sample is used as input to the function for calculating the association score between the query sample and the support set samples corresponding to the label i. and correlation weight Strengthening the focus on support set samples that are highly relevant to the query sample to measure the performance in generating a specific threshold t est Time tag prototype radius s i The weights are calculated using formulas (11) and (12).

[0066]

[0067]

[0068] Step 4.3, based on the label prototype radius s i and correlation weight Calculate a specific threshold t est The calculation formula is shown in formula (13).

[0069]

[0070] Step 4.4: Calculate the mean of the correlation scores between the query sample and R samples in other domains as the coefficient μ for evaluating the empirical importance of the general threshold. The calculation formula is shown in Formula (14).

[0071]

[0072] Step 4.5, using a specific threshold t est Correction of universal threshold t meta The threshold t of the prototype network is calculated using the formula (15).

[0073] t=μ×t meta +(1-μ)×t est (15)

[0074] Step 5: Filter out multiple intent labels corresponding to the query samples based on the threshold t of the prototype network.

[0075] Step 5.1: Divide the prototype network into inner and outer sides according to the threshold t of the prototype network, where the label prototype radius s i The region smaller than the threshold t is the inner side, and the label prototype radius s i The region greater than the threshold t is the outer region.

[0076] Step 5.2: Filter out the prototype representations of the labels within the threshold t to obtain multiple intent labels corresponding to the query sample. A schematic diagram of the prototype network is shown below. Figure 5 As shown.

[0077] Experiments were designed to verify the effectiveness of the invention. Two publicly available datasets were used in the experiments: TourSG and StanfordLU. The attributes of the experimental datasets are shown in Table 1. The TourSG dataset contains 25,751 dialogues from six independent domains of Singapore tourism information: Itinerary (It), Accommodation (Ac), Attractions (At), Food (Fo), Transportation (Tr), and Shopping (Sh). The StanfordLU dataset is a re-annotated version of the Stanford dialogue dataset, containing 8,038 dialogues from three domains: Schedule (Sc), Navigation (Na), and Weather (We).

[0078] In the experiment, each training domain of the TourSG dataset was constructed with 100 meta-tasks, each test domain with 50 meta-tasks, and the query set size was 16. Since the StanfordLU dataset has fewer domains, each training domain was constructed with 200 meta-tasks, each test domain with 50 meta-tasks, and the query set size was 32.

[0079] Table 1. Attributes of Experimental Data for Multi-Label User Intent Recognition

[0080]

[0081] The F1 score was used to evaluate the experimental results. The F1 score is the harmonic mean of precision and recall, and is calculated as shown in formulas (16), (17), and (18).

[0082]

[0083]

[0084]

[0085] Where TP represents the number of positive samples that were correctly classified as positive, FP represents the number of negative samples that were incorrectly classified as positive, and FN represents the number of positive samples that were incorrectly classified as negative.

[0086] Experimental Results: The small-sample user multi-intent recognition method with enhanced correlation calculation was used to recognize multi-label user dialogue intent on samples from TourSG and StanfordLU. The specific results of the experiment are shown in Table 2.

[0087] Table 2. F1 scores of the method of the present invention and the comparison method on the user dialogue dataset.

[0088]

[0089] Experimental results show that the method of this invention outperforms the comparative method in F1 scores in both domains, improving performance by 1.07%-8.62% compared to the prototype network-based method, indicating that the enhanced correlation calculation method can effectively improve the learning ability of the prototype network. Table 1 shows that the proportion of multi-label samples in We, Sc, and Na gradually increases, reaching 3.8%, 21.3%, and 24.6% respectively. Tests in the Sc, Na, and We domains show improvements in classification performance of 3.91%, 4.76%, and 8.62% respectively, indicating that the method has strong multi-label learning ability and is suitable for scenarios with a large number of corresponding labels for samples.

[0090] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying multiple intentions of small-sample users with enhanced correlation calculation, characterized in that... The method includes the following steps: Step 1: Use BERT to extract the embedded representations of user dialogue samples and their corresponding labels in each domain, and reconstruct the user dialogue sample set into a user multi-intent recognition task scenario under few-shot learning. Step 2: Calculate the prototype representation and prototype radius of the labels corresponding to the support set samples, and construct the prototype network. First, use the reinforcement correlation calculation function. Calculate the correlation degree between the current label and the corresponding sample in the support set, and the embedding representation y of the support set label i (i∈[1,N]). i Embedded representation of the support set samples corresponding to this label The correlation score between them is Calculate the mean Assign weights to the sample embedding representation Secondly, proportionally mixed sample embedding representation And tag embedding represents y i As a prototype representation of a label Then, the feature-level attention mechanism is used to calculate the label prototype radius. Finally, the prototype network is constructed using the query sample representation, the label prototype representation, and the label prototype radius. Step 3: Using user dialogue samples with labeled information from other domains different from the small-sample domain as prior knowledge to calculate a general threshold, firstly, the embedding representation of samples from other domains is calculated using reinforced association. Embedded representation of tags in this domain correlation between Then, the variance of the association weights between samples from other domains and each label within the domain is used. The difficulty of selecting corresponding labels for a sample is measured using the minimum relevance weight among the labels corresponding to sample r. Calculate the threshold for this sample, and then combine the thresholds from R other samples from different domains to calculate a general threshold. in This represents the association weight between the label corresponding to sample r and sample r; Step 4: Calculate a specific threshold based on the correlation between user dialogue samples in a small-sample domain. Use this specific threshold to calibrate a general threshold to calculate the prototype network threshold. First, calculate the embedding representation x0 of the query sample corresponding to the label and the embedding representation of the support set sample corresponding to the label i (i∈[1,N]) in the prototype network. correlation between Then, the correlation weights between the support set samples and the query samples under small sample conditions are used. and label prototype radius s i Calculate a specific threshold in a small sample domain Finally, the relevance score between the query sample and R samples from other domains is calculated. The mean of the values ​​is used as a coefficient to assess the empirical importance of the general threshold. Using a specific threshold t est Correction of universal threshold t meta Calculate the threshold t = μ × t for the prototype network. meta +(1-μ)×t est ; Step 5: Select a subset of labels corresponding to the query sample from the prototype network based on the prototype network threshold. First, divide the prototype network according to the prototype network threshold. Then, select the labels inside the prototype network threshold as multiple intent labels corresponding to the query sample.

Images