Clustering analysis method and equipment based on feature extraction and pseudo tag comparison, and medium

Abstract

The invention discloses a feature extraction and pseudo tag comparison-based clustering analysis method, which comprises the following steps of: after acquiring an English text sequence, converting the text sequence into an input sentence character embedding E meeting the requirements of a BERT encoder according to a BERT word segmentation device and a BERT embedding module; inputting the character embedding E into a BERT encoder module containing a shallow character representation fusion mechanism for character vector modeling to obtain an initial implicit vector H; inputting the character embedding E and the initial implicit vector H into an Electra and BERT module for local semantic extraction, and modeling the local semantic vector of the character to obtain a local semantic vector space; implicit vectors in a vector space are introduced into pseudo-label contrast clustering analysis, a pseudo label is used to evaluate and score an overall model, and contrast learning is used to perform clustering. According to the method, a better effect is achieved in an original intention recognition task. The method can be widely applied to the field of intention recognition.

Description

Technical Field

[0001] The present invention relates to the field of English intention recognition, and in particular to a clustering analysis method, device and medium based on feature extraction and pseudo-label comparison. Background Art

[0002] Understanding the user's intention is crucial to improving the overall service quality of the dialogue system. It not only helps reveal the user's potential interests, but also provides the possibility of discovering business opportunities. Traditionally, intent recognition technology relies on multi-category classification models that are trained based on sentences with clearly labeled user intentions. However, in the face of the continuous evolution of user needs, dialogue systems in real-world environments need to cope with the ever-increasing number of intent types, which is particularly evident in the need to discover new intents.

[0003] In real life, taking intelligent customer service as an example, as business expands and user needs diversify, the dialogue system needs to constantly identify new user intentions. In the e-commerce field, users may raise some special after-sales needs, or have specific consulting intentions for new products. If these new intentions are not identified in time, it will affect the user experience and the service quality of the merchant. In the financial field, users' consulting intentions for investment products are also constantly changing, and some new investment needs and risk preferences may emerge, which requires the dialogue system to quickly discover and accurately identify these new intentions.

[0004] The core of intent recognition technology lies in its innovative combination of supervised and unsupervised learning methods, which aims to automatically identify and classify unexpected user intent in dialogue systems. This technical process begins with in-depth mining of existing, labeled data sets that contain a wide range of diverse user intent examples. By analyzing this labeled data, the intent recognition algorithm can learn deep patterns and characteristics about user intent, which is a critical foundation-building process. Subsequently, the key innovative step of intent discovery is to apply the learned knowledge to unlabeled corpus, which often contains a large amount of data reflecting new user needs or expressions. It uses clustering algorithms such as K-means, DBSCAN, etc. to automatically group these unlabeled sentences according to their semantic similarity and potential purpose. The goal of clustering is to identify natural boundaries in the data to create new, previously undefined intent categories. These new categories represent new dimensions of user intent and are a direct reflection of the system's adaptability and scalability.

[0005] Despite significant progress, intent discovery still faces two core challenges that limit its practical application. Most existing methods focus on intent recognition under the framework of balanced data sets, ignoring the long-tail characteristics of intent distribution in actual situations, that is, most common intents are rich in information, while many niche or emerging intents are difficult to capture effectively due to the scarcity of samples. Taking the intelligent customer service of e-commerce platforms as an example, in addition to user information and conversation information, purchased product information will also affect the reasoning of user intent, and different user habits will also affect different intents in the same paragraph. In addition to long-tail information, the emergence of new intents also requires the intelligent customer service system to be able to discover and accurately identify them in a timely manner in order to provide effective help to users. Also taking the intelligent customer service of e-commerce platforms as an example, with the launch of new promotional activities and product categories, users may have some questions and intentions that have never appeared before. When a new product is launched, users may ask questions about the special functions, usage methods, compatibility, etc. of the product. Researchers use advanced attention mechanisms and enhanced memory models in the MTP-CLNN method, but do not absorb specific information in the conversation, resulting in poor clustering results. In view of the above challenges, it is particularly urgent to explore new strategies suitable for the task of imbalanced new intention discovery, aiming to narrow the gap between theoretical research and practical applications and better meet the dynamically changing real-world needs. Summary of the invention

[0006] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a clustering analysis method, device and medium based on feature extraction and pseudo-label comparison.

[0007] The first technical solution adopted by the present invention is:

[0008] A cluster analysis method based on feature extraction and pseudo-label comparison includes the following steps:

[0009] S1. Obtain a text sequence and convert it into an input sentence character embedding E that meets the requirements of the BERT encoder.

[0010] S2, input the character embedding E into the BERT encoder module containing the shallow character representation fusion mechanism for character vector modeling to obtain the latent vector H;

[0011] S3, calculate the attention score according to the latent vector H, assign attention weights to each character and its context characters, and then extract and integrate the features associated with each label based on these weighted features to obtain a refined label encoding representation;

[0012] S4, performing preliminary label assignment on some unlabeled labels through a pseudo-labeling method, and then obtaining an encoded representation for each potential entity span for each predefined label;

[0013] S5. Use contrastive learning to evaluate the same label category and calculate whether the classification threshold matches the given span feature to optimize the label matching accuracy. At the same time, cluster the entire word segmentation embedding space and transform it into a more compact and highly discriminative representation to obtain the final target entity.

[0014] Furthermore, the step S1 comprises:

[0015] After obtaining the text sequence, the text sequence is input into the BERT word segmenter for format conversion to obtain the input sentence X = {x1, x2, ..., x t ,…,x n}, where n is the length of the input sentence X;

[0016] Input the input sentence X into the BERT embedding module for character embedding modeling, and obtain the character embedding E = {E1, E2, …, E t ,…,E n}.

[0017] Furthermore, the step S2 comprises:

[0018] Input the character embedding E into the first Transformer encoding layer of the BERT encoder for implicit vector encoding modeling to obtain the implicit vector H of the first layer 1 ;

[0019] The latent vector H obtained by the b-1th Transformer encoding layer in the BERT encoder b-1 Input to the b-th Transformer encoding layer to obtain the implicit vector of the b-th layer The formula is as follows:

[0020] H b = Transformer_Encoder(H b-1 ) (1)

[0021] Among them, H 0 =E, b represents the number of layers, counting from 0; t is the total number of layers, the default is 5; Transformer_Encoder() represents the encoder operation in the Transformer architecture.

[0022] Furthermore, the step S3 comprises:

[0023] According to the latent vector H b-1 Perform attention score calculation to obtain the attention score S(X) of each character and the context character. The calculation formula is as follows:

[0024] Query = Wq E (2)

[0025] Key=W k H b-1 (3)

[0026]

[0027] Where W q ,W k and U are both trainable parameters; represents the self-attention score of layer b; N h is the total number of heads of multi-head attention, head is the number of attention heads, is the query vector in the self-attention matrix, is the key vector in the attention matrix, and d is the dimension of the self-attention head;

[0028] Use the attention score S(X) as a weight to the latent vector H b After updating, a more refined label encoding representation is obtained. The specific formula is as follows:

[0029] H b =H b-1 +S(X) b ⊙H m (5)

[0030] Among them, the new m=b-B+M, B is the number of layers of the BERT model, M∈[1,B], represents the current number of layers calculated, the default value is 5; ⊙ represents the multiplication of the corresponding positions;

[0031] The implicit vector corresponding to the [CLS] mark The input is sent to the fully connected linear classification layer of the target-guided keyword extraction module for the intent classification task. After mapping, the probability distribution P(Y|X) for the intent category is obtained. The calculation formula is as follows:

[0032]

[0033] Among them, W and bias are both trainable parameters;

[0034] The intent category corresponding to the maximum value of P(Y|X) is obtained as the final prediction result; according to the cross entropy loss function, the distance between P(Y|X) and the target distribution is calculated to obtain the current loss value Loss.

[0035] Furthermore, the step S4 comprises:

[0036] The sample is classified and predicted by the model pre-trained in step S3 to generate pseudo labels, and the relaxed optimal transfer method is applied, where Q and P represent the pseudo label matrix and classifier prediction respectively, and the allocation of these pseudo labels is optimized in combination with the specifically defined cost matrix. The objective function is constructed according to formula (7), which aims to improve the classification performance and generalization ability of the model under the condition of limited labeled data:

[0037] min<Q,log P> +λH′(Q)(7)

[0038] Among them, the function H′ represents entropy regularization and λ is a scalar factor.

[0039] Furthermore, in order to solve the problem of unreliable pseudo labels caused by long-tail distribution and unknown classes, an improved strategy is introduced:

[0040] By introducing the relaxed optimal transfer (ROT) with soft constraints, the KL divergence constraint is used to make the prediction distribution of categories more uniform. This can not only effectively deal with the challenge of uneven data distribution and prevent the model from learning simplified or degenerate solutions, but also flexibly adapt to the actual situation of imbalanced number of different categories. Compared with strict equality constraints, it is more flexible and practical. In this way, the pseudo label matrix U is obtained;

[0041] The specific mathematical expression of the relaxed optimal transmission with soft constraints is given by formula (8):

[0042]

[0043] Where λ1 and λ2 are Lagrange multipliers; D KL is the KL divergence; K is the total number of categories; β is the probability distribution of all intent categories.

[0044] The quality of the training signal is improved by classifying pseudo labels according to their distribution characteristics and quality, distinguishing clean from noisy labels. The contrastive loss method is used to promote representation learning that is close between similar samples and far between heterogeneous samples, which helps to form more discriminative feature clusters in noisy environments. In addition, to further deal with the noise problem, distribution-aware regularization is introduced. This regularization technique dynamically adjusts the learning rate or sample weights based on the loss size of the sample and the prior distribution of the class to which it belongs, so as to align the sample selection with the true class distribution.

[0045] Furthermore, the step S5 comprises:

[0046] High-quality pseudo-labels use the contrastive loss principle to guide the selection process of positive sample pairs by integrating these pseudo-label information. Specifically, we take the sample pairs of the same category in the set as positive pairs. Based on such a distance metric, the samples in the hypothesis share similar semantic intent or topic features because they are close to each other in the high-dimensional vector space.

[0047] In addition, a mechanism is designed to dynamically adjust the weight of the positive association class contrast loss between sample pairs; this mechanism considers the prediction confidence score q of each sample i and q j (represents sample x i and x j The classification confidence) is calculated to obtain the adaptive weight w ij =q i ·q j ; Through this method, sample pairs with high confidence will contribute more to the loss function, as shown in formula (9)(10):

[0048]

[0049] P(i)={p|(p∈U)∩(c i =c p )} (10)

[0050] Among them, P(i) represents the i The indices of instances that share the same label, is a hyperparameter.

[0051] Furthermore, the utterances are mapped into an embedding vector space, which aims to capture the semantic information of the sentences so that utterances with similar intentions or meanings are close in the space;

[0052] Using the inner product as the similarity metric, find the K closest neighbors of each utterance Xi in the space to form its neighborhood Ni; for each utterance Xi in the mini-batch, further randomly select a neighbor X'i from its neighborhood Ni, and find a direct "approximate" sample for each utterance so that the model can learn the similarity between them; pool all data samples to construct an adjacency matrix A' of size 2M×2M; this matrix is ​​used to encode the relationship between sample pairs. If two samples are from the neighborhood with the same intent label, they are marked as a positive relationship; otherwise, they are marked as a negative relationship; such a design encourages the model to distinguish between positive and negative sample pairs in the contrastive learning framework, thereby learning better semantic representations; the loss function is shown in (11):

[0053]

[0054] Among them, |C i| indicates that the words have a positive relationship, and sim(·,·) indicates the cosine similarity;

[0055] Combine all similarity losses as the final loss function.

[0056]

[0057] Among them, λ is the weight; Calculate the contrast loss weight for the positive association class mentioned in formula (9).

[0058] The second technical solution adopted by the present invention is:

[0059] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement a clustering analysis method based on feature extraction and pseudo-label comparison as described above.

[0060] The fourth technical solution adopted by the present invention is:

[0061] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a clustering analysis method based on feature extraction and pseudo-label comparison as described above.

[0062] The fifth technical solution adopted by the present invention is:

[0063] A computer program product or a computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method.

[0064] The beneficial effects of the present invention are as follows: the present invention is the first to identify known intents and inferred new intents in long-tail data distribution. The method alleviates the problem of label knowledge being dispersed by the original text when the label knowledge is integrated with the entire text through iterative interaction of pseudo-label generation and representation learning, and also reduces the training and reasoning time of the model. It promotes feature representation that is conducive to clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 It is a schematic diagram of the feature detector process of the intent recognition method based on feature extraction and pseudo-label comparative clustering analysis in an embodiment of the present invention.

[0067] Figure 2 It is an overall schematic diagram of the intent recognition method based on feature extraction and pseudo-label comparative clustering analysis in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0069] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0070] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0071] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0072] In the process of English intent recognition, facing massive and complex English text data, how to accurately extract features that can accurately reflect user intent is a major challenge. Existing methods often have limitations in the feature extraction link and may not be able to fully cover the multi-dimensional information in the text, such as semantics, grammar, and vocabulary, resulting in deviations in the understanding of intent. For example, for some English expressions with metaphorical and punning meanings, traditional methods may miss key features.

[0073] At the same time, in the cluster analysis stage, the diversity of data and the interference of noise make it difficult to accurately classify intent. Traditional clustering methods may mistakenly classify texts with similar pseudo-labels but different actual intents into one category due to the lack of an effective comparison mechanism, or classify texts with the same intent but slightly different expressions into different categories, which seriously affects the effect of intent recognition. The cluster analysis method based on feature extraction and pseudo-label comparison of the present invention aims to solve these key problems in the field of English intent recognition and improve the accuracy and efficiency of intent recognition through innovative technical means.

[0074] Example 1

[0075] At present, with the emergence of various intelligent customer services, the expansion of business and the diversification of user needs, the dialogue system needs to continuously identify new user intentions. In the field of e-commerce, users may have some special after-sales needs, or have specific consulting intentions for new products. If these new intentions cannot be identified in time, it will affect the user experience and the service quality of the merchant. In the financial field, users' consulting intentions for investment products are also constantly changing, and some new investment needs and risk preferences may emerge, which requires the dialogue system to be able to quickly discover and accurately identify these new intentions. However, most existing methods focus on processing intent recognition under the framework of balanced data sets, ignoring the long-tail characteristics of intent distribution in actual situations, that is, most common intent data are rich, while many niche or emerging intents are difficult to effectively capture due to scarce samples. In view of the shortcomings of the prior art, this embodiment provides a clustering analysis method based on feature extraction and pseudo-label comparison. The method uses the pre-trained language model BERT as the model backbone, designs a local semantic extraction module based on the self-attention mechanism, and supplements the local semantic representation of the context to the implicit vector of the pre-trained language model, thereby assisting the model understanding and improving the effect of intent recognition, which has broad practical value. Figure 2 As shown, the method comprises the following steps:

[0076] S1. Perform word segmentation on the input text to obtain the word segmentation sequence, which is then input into the pre-trained BERT model to extract its deep hidden state representation to provide rich semantic features for subsequent entity recognition and classification tasks;

[0077] like Figure 1 As shown in the figure, in order to identify the main intention of the user in asking questions, relevant English text is extracted from the conversation between the customer service and the user. After obtaining the English text sequence, the English text sequence is input into the BERT word segmenter for format conversion to obtain the input sentence X, which is expressed as X = {x1, x2, ..., x t ,…,x n}, where n is the length of the input sentence X, x t ∈[1,N v ],N v =21183 represents the length of the vocabulary in the BERT word segmenter; then the input sentence X is input into the BERT embedding module for character embedding modeling, and the character embedding E of the input sentence is obtained, which is expressed as E={E1,E2,…,E t ,…,E n}, where E t ∈R d , R d represents a one-dimensional real number vector with dimension d=768.

[0078] S2. Input the character embedding E into the BERT encoder module containing the shallow character representation fusion mechanism for character vector modeling to obtain the initial latent vector H.

[0079] like Figure 1 As shown, the character embedding E is input into the first Transformer encoding layer of the BERT encoder for implicit vector encoding modeling to obtain the implicit vector H of the first layer. 1 ; Then the implicit vector H obtained by the i-1th Transformer encoding layer in the BERT encoder i-1 Input to the i-th Transformer encoding layer, perform implicit vector encoding modeling according to calculation formula (1), and obtain the i-th layer's implicit vector H i ;

[0080]

[0081] Among them, Encoder represents the operation performed by the Transformer encoding layer, and i represents the i-th layer. Finally, the implicit vector of the last layer is called the initial implicit vector, which is expressed as H = {H1, H2, ..., H t ,…,H n}, where H t ∈R d , R d represents a one-dimensional real number vector with dimension d=768.

[0082] S3, input the hidden state representation into the target-guided keyword extraction module containing the multi-head attention mechanism, and assign attention weights to each character and its context characters by performing attention distribution calculation. Then, based on these weighted features, the features associated with each label are extracted and integrated to obtain a refined label encoding representation;

[0083] like Figure 1 As shown, E and H b-1 The attention score is calculated by inputting it into the calculation unit composed of the multi-head attention mechanism of the target-guided keyword extraction module to obtain the attention importance score of each character and the context character, expressed as S(X), where S(X) = {S(X)1, S(X)2, …, S(X) t ,…,S(X) n}, the calculation process is shown in formula (2)(3)(4);

[0084] Query = W q E (2)

[0085]

[0086] Among them, W q ,W k and U are both trainable parameters and belong to R d×d , R d×d represents a two-dimensional real matrix, where R represents a real number, N h is the total number of heads of multi-head attention, and Key is the key vector for calculating attention.

[0087] Then, the preliminary implicit vector of the b-th layer is obtained by formula (1), and then is used as the weight of to obtain the weighted ; then the weighted is added to to update the implicit vector. The calculation process is shown in formula (5):

[0088] H b =H b-1 +S(X) b ⊙H m (5)

[0089] in R d represents a one-dimensional real vector with dimension d, m=b-B+M, and ⊙ represents the multiplication of corresponding positions.

[0090] The implicit vector corresponding to the [CLS] mark The input is sent to the fully connected linear classification layer of the target-guided keyword extraction module for the intent classification task. After mapping, the probability distribution P(Y|X) for the intent category is obtained, expressed as P(Y|X). The calculation process is shown in formula (6);

[0091]

[0092] Where W∈R Ni×d , R Ni×d represents a two-dimensional real matrix, Indicates that the dimension is N i A one-dimensional real vector, and W and bias are both trainable parameters, N i The number of categories representing intent.

[0093] The intent category corresponding to the maximum value of P(Y|X) is obtained, and the current loss value Loss is obtained.

[0094] S4, pseudo-labeling method performs preliminary label assignment on some unlabeled labels, and then obtains an encoded representation for each predefined label for each potential entity span;

[0095] like Figure 2 As shown in the figure, the samples are classified and predicted by the model pre-trained in S3, pseudo labels are generated, and the relaxed optimal transfer method is applied, where Q∈R and P∈R represent the pseudo label matrix and classifier prediction respectively. The allocation of these pseudo labels is optimized in combination with the specifically defined cost matrix. The objective function is constructed according to formula (7), which aims to improve the classification performance and generalization ability of the model under the condition of limited labeled data.

[0096] min<Q,log P> +λH(Q)(7)

[0097] Among them, the function H represents entropy regularization and λ is a scalar factor.

[0098] In order to solve the problem of unreliable pseudo-labels caused by long-tail distribution and unknown classes, we introduced an improved strategy. By introducing relaxed optimal transfer (ROT) with soft constraints, especially using KL divergence constraints to make the predicted distribution of categories more uniform. This can not only effectively deal with the challenge of uneven data distribution and prevent the model from learning simplified or degenerate solutions, but also flexibly adapt to the actual situation of imbalanced number of different categories. It is more flexible and practical than strict equality constraints. This method obtains the pseudo-label matrix U. The specific mathematical expression of ROT is given by formula (8):

[0099]

[0100] The quality of the training signal is improved by classifying pseudo labels according to their distribution characteristics and quality, distinguishing clean from noisy labels. The contrastive loss method is used to promote representation learning that is close between similar samples and far between heterogeneous samples, which helps to form more discriminative feature clusters in noisy environments. In addition, to further deal with the noise problem, distribution-aware regularization is introduced. This regularization technique dynamically adjusts the learning rate or sample weights based on the loss size of the sample and the prior distribution of the class to which it belongs, so as to align the sample selection with the true class distribution.

[0101] S5, high-quality pseudo-labels use the contrast loss principle to guide the selection process of positive sample pairs by integrating these pseudo-label information. Figure 2 As shown in Figure 1, in practice, we identify the sample pairs of the same category in the set as positive pairs. Based on such a distance metric, the samples in the hypothesis share similar semantic intent or topic features because they are close to each other in the high-dimensional vector space. In addition, we design a mechanism to dynamically adjust the weight of the positive association class contrast loss between sample pairs. This mechanism considers the prediction confidence score q of each sample. i and q j (represents sample x i and x j The classification confidence) is calculated to obtain the adaptive weight w ij =q i ·q j In this way, sample pairs with high confidence will contribute more to the loss function. The formula is shown in (9) (10):

[0102]

[0103] P(i)={p|(p∈U)∩(c i =c p )} (10)

[0104] Where P(i) represents the i The indices of instances that share the same label, is a hyperparameter.

[0105] In addition, the utterances are mapped into an embedding vector space. This space aims to capture the semantic information of the sentence, so that utterances with similar intentions or meanings are close in the space. Using the inner product as a similarity measure, we can find out the similarity of each utterance. □ The K closest neighbors in this space constitute its neighborhood N i For each utterance X in a mini-batch i , from its neighborhood N i Further randomly select a neighbor X' i, find a direct "approximate" sample for each utterance so that the model can learn the similarity between them. All data samples are pooled to construct an adjacency matrix A' of size 2M×2M. This matrix is ​​used to encode the relationship between sample pairs. If two samples are from the neighborhood with the same intent label, they are marked as a positive relationship; otherwise, they are marked as a negative relationship. Such a design encourages the model to distinguish between positive and negative sample pairs under the contrastive learning framework, thereby learning better semantic representations. The loss function is shown in (11):

[0106]

[0107] Where |C i | indicates that the utterances have a positive relationship, and sim(·,·) indicates the cosine similarity.

[0108] Combine all similarity losses as the final loss function.

[0109]

[0110] In summary, the embodiment of the present invention uses a given English dataset to design a clustering analysis method based on feature extraction and pseudo-label comparison, so as to further integrate shallow characters and local semantic knowledge into the character vector of the pre-trained language model, and use pseudo-labels to improve the quality of model intent discovery.

[0111] When conducting experiments, we used the Banking dataset and the Stackoverflow dataset. The results of the experiments are shown in Table 1 below:

[0112] Table 1: Comparative experiment

[0113] Banking Stackoverflow MTP-CLNN 84.11％ 79.68％ USNID 86.53％ 77.32％ ImbaNID 86.66％ 81.47％ This method 87.13％ 83.09％

[0114] Compared with the prior art, the present invention has at least the following advantages and beneficial effects:

[0115] (1) The present invention provides a clustering analysis method based on feature extraction and pseudo-label comparison, which is simple to implement and does not introduce other external knowledge sources, and has strong migration capabilities.

[0116] (2) The local semantic extraction module designed in the present invention can extract more features of the characters in the original text sequence and alleviate the influence of irrelevant factors, thereby further improving the accuracy of intent recognition of the customer service dialogue model and improving the quality of model prediction.

[0117] (3) The present invention provides a clustering analysis method based on feature extraction and pseudo-label comparison, which has good recognition effect for different types of intent recognition tasks and has excellent universality.

[0118] Example 2

[0119] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 2 A clustering analysis method based on feature extraction and pseudo-label comparison is shown.

[0120] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0121] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0122] Since the electronic device is an electronic device corresponding to the method for clustering analysis based on feature extraction and pseudo-label comparison in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0123] Example 3

[0124] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 2 A clustering analysis method based on feature extraction and pseudo-label comparison is shown.

[0125] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0126] Since the storage medium is a storage medium corresponding to the feature extraction and pseudo-label comparison clustering analysis method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0127] Example 4

[0128] In some possible implementations, various aspects of the method of the embodiment of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a clustering analysis method based on feature extraction and pseudo-label comparison according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0129] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0130] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0131] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A clustering analysis method based on feature extraction and pseudo-label comparison, characterized in that: The following steps are involved: S1. Obtain a text sequence and convert it into an input sentence character embedding E that meets the requirements of the BERT encoder. S2, input the character embedding E into the BERT encoder module containing the shallow character representation fusion mechanism for character vector modeling to obtain the latent vector H; S3, calculate the attention score according to the latent vector H, assign attention weights to each character and its context characters, and then extract and integrate the features associated with each label based on these weighted features to obtain a refined label encoding representation; S4, performing preliminary label assignment on some unlabeled labels through a pseudo-labeling method, and then obtaining an encoded representation for each potential entity span for each predefined label; S5. Use contrastive learning to evaluate the same label category and calculate whether the classification threshold matches the given span feature to optimize the label matching accuracy. At the same time, cluster the entire word segmentation embedding space and transform it into a more compact and highly discriminative representation to obtain the final target entity.

2. The cluster analysis method based on feature extraction and pseudo-label comparison according to claim 1, characterized in that: The step S1 comprises: After obtaining the text sequence, the text sequence is input into the BERT word segmenter for format conversion to obtain the input sentence X = {x1, x2, ..., x t ,…,x n }, where n is the length of the input sentence X; Input the input sentence X into the BERT embedding module for character embedding modeling to obtain the character embedding of the input sentence E={E1,E2,…,E t ,…,THE n }。 3. The cluster analysis method based on feature extraction and pseudo-label comparison according to claim 1, characterized in that: The step S2 comprises: Input the character embedding E into the first Transformer encoding layer of the BERT encoder for implicit vector encoding modeling to obtain the implicit vector H of the first layer 1 ; The latent vector H obtained by the b-1th Transformer encoding layer in the BERT encoder b-1 Input to the b-th Transformer encoding layer to obtain the implicit vector of the b-th layer The formula is as follows: H b =Transformer_Encoder(H b-1 ) (1) Among them, H 0 =E, b represents the number of layers, counting from 0; t is the total number of layers; Transformer_Encoder() represents the encoder operation in the Transformer architecture.

4. The cluster analysis method based on feature extraction and pseudo-label comparison according to claim 3 is characterized in that: The step S3 comprises: According to the latent vector H b-1 Perform attention score calculation to obtain the attention score S(X) of each character and the context character. The calculation formula is as follows: Query=W q E (2) Key=W k H b-1 (3) Where W q ,W k and U are both trainable parameters; represents the self-attention score of layer b; N h is the total number of heads of multi-head attention, head is the number of attention heads, is the query vector in the self-attention matrix, is the key vector in the attention matrix, and d is the dimension of the self-attention head; Use the attention score S(X) as a weight to the latent vector H b After updating, a more refined label encoding representation is obtained. The specific formula is as follows: H b =H b-1 +S(X) b ⊙H m (5) Among them, the new m=b-B+M, B is the number of layers of the BERT model, M∈[1,B], represents the current number of layers calculated; ⊙ represents the multiplication of the corresponding positions; The implicit vector corresponding to the [CLS] mark The input is sent to the fully connected linear classification layer of the target-guided keyword extraction module for the intent classification task. After mapping, the probability distribution P(Y|X) for the intent category is obtained. The calculation formula is as follows: Among them, W and bias are both trainable parameters; The intent category corresponding to the maximum value of P(Y|X) is obtained as the final prediction result; according to the cross entropy loss function, the distance between P(Y|X) and the target distribution is calculated to obtain the current loss value Loss.

5. The cluster analysis method based on feature extraction and pseudo-label comparison according to claim 4 is characterized in that: The step S4 comprises: The sample is classified and predicted by the model pre-trained in step S3 to generate pseudo labels, and the relaxed optimal transfer method is applied, where Q and P represent the pseudo label matrix and classifier prediction respectively, and the allocation of these pseudo labels is optimized in combination with the specifically defined cost matrix. The objective function is constructed according to formula (7), which aims to improve the classification performance and generalization ability of the model under the condition of limited labeled data: min<Q,log P> +λH ′ (Q)(7) where function H ′ represents entropy regularization and λ is a scalar factor.

6. The cluster analysis method based on feature extraction and pseudo-label comparison according to claim 5, characterized in that: In order to solve the problem of unreliable pseudo labels caused by long-tail distribution and unknown classes, an improved strategy is introduced: By introducing a relaxed optimal transfer with soft constraints, the KL divergence constraint is used to make the prediction distribution of categories more uniform; The pseudo label matrix U is obtained by this method; The specific mathematical expression of the relaxed optimal transmission with soft constraints is given by formula (8): Where λ1 and λ2 are Lagrange multipliers; D KL is the KL divergence; K is the total number of categories; β is the probability distribution of all intent categories.

7. The cluster analysis method based on feature extraction and pseudo-label comparison according to claim 1, characterized in that: The step S5 comprises: A mechanism is designed to dynamically adjust the weight of the positive association class contrast loss between sample pairs; this mechanism considers the prediction confidence score q of each sample i and q j , calculate the adaptive weight w ij =q i ·q j ; Through this method, sample pairs with high confidence will contribute more to the loss function, as shown in formula (9)(10): P(i)={p|(p∈U)∩(c i =c p )} (10) Among them, P(i) represents the i The indices of instances that share the same label, is a hyperparameter.

8. The cluster analysis method based on feature extraction and pseudo-label comparison according to claim 7, characterized in that: Mapping utterances into an embedding vector space that aims to capture the semantic information of sentences so that utterances with similar intent or meaning are close in the space; Using the inner product as the similarity measure, find the similarity of each utterance X i The K closest neighbors in this space constitute its neighborhood N i ; For each utterance X in a mini-batch i , from its neighborhood N i Further randomly select a neighbor X' i , find a direct "approximate" sample for each utterance so that the model can learn the similarity between them; pool all data samples to construct an adjacency matrix A' of size 2M×2M; this matrix is ​​used to encode the relationship between sample pairs. If two samples are from the neighborhood with the same intent label, they are marked as a positive relationship; otherwise, they are marked as a negative relationship; such a design encourages the model to distinguish between positive and negative sample pairs under the contrastive learning framework, thereby learning better semantic representation; the loss function is shown in (11): Among them, |C i | indicates that the words have a positive relationship, and sim(·,·) indicates the cosine similarity; Combine all similarity losses as the final loss function. Among them, λ is the weight; Compute the contrast loss weight for the positive association class.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

Images