An intelligent question and answer semantic representation method fusing contrastive learning and clustering

Abstract

The application discloses an intelligent question and answer semantic representation method fusing contrast learning and clustering, characterized in that an open-source bert pre-training model is loaded, and the bert pre-training model is trained by using contrast learning to obtain a general field bert semantic representation model; a clustering algorithm is used to carry out quality inspection on a knowledge base, and standard questions without repetition and intersection are found and screened out; the general field bert semantic representation model is further trained by using contrast clustering on a data set in a business field to obtain a business field bert semantic representation model; and efficient sample semantic representation and category semantic representation learning in the field of intelligent question and answer are realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent question answering, and in particular to an intelligent question answering semantic representation method that integrates contrastive learning and clustering. Background Technology

[0002] With the development of artificial intelligence, chatbots, intelligent assistant robots, and other products have been widely used in people's daily lives. Intelligent question answering, a core function of robots, has seen significant improvements in performance thanks to continuous updates in natural language processing (NLP) technology. BERT, as a general-purpose pre-trained model, is widely used in NLP due to its excellent performance. However, as a semantic representation method, BERT's semantic space is anisotropic. This anisotropy leads to uneven distribution of word vectors; high-frequency words are denser and closer to the origin, while low-frequency words are sparser and farther from the origin. Therefore, the original BERT cannot be directly used for semantic representation.

[0003] To address the anisotropy issue in BERT semantic representation, BERT-flow uses normalizing flows to transform the BERT sentence vector distribution into a Gaussian distribution, resulting in a smooth, isotropic distribution space. However, the normalizing flows used in this method are complex and not well-suited for business applications.

[0004] Therefore, a highly targeted and widely applicable semantic representation method for intelligent question answering is needed. Summary of the Invention

[0005] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies and provide an intelligent question-answering semantic representation method that integrates contrastive learning and clustering. Contrastive learning has been continuously explored in the natural language processing field. SimCSE, for example, uses contrastive learning to narrow the distance between similar sentences and widen the distance between dissimilar sentences to simply address BERT's anisotropy. While SimCSE can optimize BERT's semantic representation performance through contrastive learning, as a sample-pair-based training method, it can only learn semantic information at the sample-pair level and cannot learn high-level semantic information at the category level. However, the knowledge base of intelligent question answering is organized in a standard question-similar question format, where each standard question represents a category. This requires the model to learn both sample-pair-level semantic information and high-level category-level semantic information. Therefore, SimCSE cannot meet the requirement of high-level category-level semantic information in intelligent question answering.

[0006] To achieve better question-answering results, the semantic representation method used must learn both the semantic information of sample pairs and the high-level semantic information of categories. This invention provides an intelligent question-answering semantic representation method that integrates contrastive learning and clustering, as detailed below:

[0007] Step 1: Collect sample pairs of data from the general domain, load the open-source BERT pre-trained model, and use contrastive learning to train the BERT pre-trained model to obtain the general domain BERT semantic representation model.

[0008] Step 2: Use clustering algorithms to perform quality checks on the knowledge base;

[0009] Step 3: On the dataset of the business domain, use contrastive clustering to continue training the general domain BERT semantic representation model to obtain the business domain BERT semantic representation model.

[0010] Step 1 includes the following specific steps:

[0011] Step 1-1: Collect sample pairs of data from a general domain as training sample pairs;

[0012] Steps 1-2: Build the BERT model and load the corresponding open-source pre-trained weights of the BERT model to obtain the BERT pre-trained model;

[0013] Steps 1-3: Using the general domain dataset collected in Step 1, continue training the BERT pre-trained model with contrastive loss to obtain the general domain BERT semantic representation model.

[0014] In steps 1-2, the pre-trained weights are from the Roberta-WWM-Base model. In steps 1-3, the BERT pre-trained model is optimized using a contrastive loss function to obtain a general-domain BERT semantic representation model. The contrastive loss function is:

[0015]

[0016] Where I indicates that the loss is a contrastive loss, M represents the number of sentence pairs in a batch, i and j represent the index numbers of the batches to which the two sets of sentences belong, and 1 and 2 represent the numbers of different sentences belonging to the same index. This represents the positive sample pair constructed from the sentence located at index i. τ represents the negative sample pair constructed from the sentence at index i and the sentence at index j, and τ is the temperature control coefficient of the control model.

[0017] Step 2 includes clustering the questions in the knowledge base based on similarity, identifying and filtering out non-overlapping standard questions, which includes the following specific steps:

[0018] Step 2-1: Encode all questions in the knowledge base using the general domain BERT semantic representation model to obtain the sentence vector representation of each question;

[0019] Step 2-2: Perform fast clustering using the cosine similarity of sentence vectors. When the cosine similarity of the sentence vectors of two questions is greater than a preset threshold, they are set as similar, and all sentence vectors are divided into more than one category.

[0020] Steps 2-3: Detect whether all sentence vectors in each category belong to the same standard question. If the sentence vectors of a category belong to multiple standard questions and exceed the preset proportion, then merge the questions under that category.

[0021] In step 3, sample pairs are sampled from non-repeating standard questions, and a BERT semantic representation model for the business domain is trained using a joint loss function.

[0022] In step 3, the sentence vector μ for each category k is represented using the weights of the MLP. k Using the t-distribution to calculate the original problem x j The probability of being classified into the k-th category is:

[0023]

[0024] Where K represents the number of standard questions in the knowledge base, and k represents x j The category index to which it belongs, k′ represents one of the category indices in terms of the number of standard queries, μ k ,k∈{1,…,K} represents the sentence vector for each category, e j =ψ(x j ) represents each problem x j The sentence vector, α represents the degrees of freedom of the t distribution, and finally an auxiliary probability is used as the target probability distribution, which is expressed as:

[0025]

[0026] in The sum of the probabilities of all questions representing the center of the k-th category is f. k′ The sum of probabilities representing all questions at the k′-th category center;

[0027] By using KL divergence to bring the class probability distribution closer to the target distribution, the clustering loss is obtained as follows:

[0028]

[0029] The term C is used to distinguish contrast loss, indicating that the loss is a clustering loss;

[0030] The joint loss is expressed as:

[0031]

[0032] Where η represents the weight of the contrast loss.

[0033] In steps 1-3, the number of sentence pairs is 256, the temperature control coefficient τ is 0.5, and in step 3, η is 10.

[0034] The beneficial effects achieved by this invention are as follows: By training a BERT pre-trained model on general domain sample data based on contrastive learning, the anisotropy problem in the semantic space of the BERT pre-trained model is solved; by clustering the questions in the knowledge base through knowledge base quality inspection, the noise problem of repeated and overlapping standard questions in the knowledge base is solved; by continuing to train the BERT general domain semantic representation model on business domain sample data based on contrastive clustering, the problem that the BERT general domain semantic representation model can only learn semantic information at the sample pair level and cannot learn high-level semantic information at the category level is solved. Attached Figure Description

[0035] Figure 1 A system flowchart illustrating an exemplary embodiment of the present invention;

[0036] Figure 2 This is a flowchart of the knowledge base quality inspection process of the present invention;

[0037] Figure 3 This is a flowchart of the BERT comparison clustering training process for the business domain of this invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and exemplary embodiments:

[0039] As attached Figure 1 The following is a method for intelligent question answering semantic standards based on fusion contrastive learning and clustering:

[0040] Step 1, BERT pre-trained model contrastive learning training, specifically involves collecting sample pairs of data from a general domain, loading an open-source BERT pre-trained model, and training the BERT pre-trained model using contrastive learning; resulting in a general domain BERT semantic representation model. The specific steps include the following:

[0041] Step 1-1: Collect a Chinese natural language inference dataset as training sample pairs. The dataset is characterized by similar questions having less lexical overlap, which enables the model to learn better semantic information.

[0042] Steps 1-2: Select Roberta-WWM-Base as the BERT pre-trained model. Compared with the traditional BERT, it is trained with a larger dataset and full-word masking, achieving better results.

[0043] Steps 1-3: The training samples from Step 1-1 are input into the BERT pre-trained model from Step 1-2 to obtain sentence vectors. These sentence vectors are represented by 768-dimensional vectors. Finally, a contrastive loss function is used to minimize the distance between sentence vectors of positive sample pairs and maximize the distance between sentence vectors of negative sample pairs. The contrastive loss function is:

[0044]

[0045] Where I indicates that the loss is a contrastive loss, M represents the number of sentence pairs in a batch, and the value of M should be as large as possible. However, due to memory limitations in the experiment, after practical comparison and adjustment, the final value of M is 256. i and j represent the index numbers of the batches to which the two sets of sentences belong, respectively, and 1 and 2 represent the numbers of different sentences belonging to the same index. This represents a positive sample pair consisting of the i-th sentence and its similar sentences, indicating that the sample pair has a "premise-hypothesis" similarity relationship. Negative sample pairs represent two sample pairs that do not belong to the "premise-assumption" relationship. τ is the temperature control coefficient, representing the model's attention to difficult samples, enabling the algorithm to automatically discover difficult samples, thereby accelerating model convergence. In the experiment, a value of 0.5 was used. This value can increase the model's attention to difficult negative sample pairs, thereby improving the model's convergence speed and performance. There is no easily predictable proportional relationship between this value and performance, and 0.5 is an optimal assignment suitable for application derived from practice. Simply increasing or decreasing this assignment will cause performance loss in the system.

[0046] Step 2, knowledge base quality inspection, specifically involves clustering the questions in the knowledge base based on similarity to identify and filter out standard questions that do not overlap.

[0047] like Figure 2 As shown, the specific steps include the following:

[0048] Step 2-1: Knowledge base question sentence vector encoding. Using the general domain BERT semantic representation model, all questions in the knowledge base are encoded to obtain the sentence vector for each question;

[0049] Step 2-2: Fast clustering of sentence vectors in the knowledge base. Fast clustering is performed by using the cosine similarity of sentence vectors. When the cosine similarity of the sentence vectors of two questions is greater than 0.94, the sample pair is considered to be semantically consistent. Finally, multiple semantically consistent samples are clustered into one class.

[0050] Step 2-3: Knowledge Base Standard Question Duplication Detection. This step checks whether all questions in each subcategory from Step 2-2 belong to the same standard question. If more than 1 / 3 of the questions are inconsistent, it indicates that the standard questions belonging to that subcategory overlap, and the questions in that subcategory need to be merged.

[0051] Step 3: Comparative clustering training of the general domain BERT semantic representation model. Specifically, using sample pairs of non-repeating standard questions selected from the knowledge base, the comparative clustering loss is used to continue training the general domain BERT semantic representation model.

[0052] like Figure 3 As shown, the specific steps include the following:

[0053] Step 3-1: Sample training samples are obtained by sampling from the standard question and similar question obtained in Step 2. The standard questions of the sample pairs in the same batch cannot be the same.

[0054] Step 3-2: Design the contrastive clustering loss. Unlike the contrastive loss function, the clustering loss focuses more on high-level semantic concepts and groups samples with similar high-level semantic concepts into the same category. Since there are 500 standard questions in the experiment, the number of categories K is set to 500. This model uses the weights of an MLP to approximate the category vector μ. k Each category vector has a length of 500; and each question x j The sentence vector is represented as e j =ψ(x j If x is a 768-dimensional sentence vector, then the original problem x can be calculated using the t-distribution. j The probability of being classified into the k-th category is:

[0055]

[0056] Here, K is the maximum value that k can take. For example, if the number of standard questions is 10, then K = 10, and the range of k is 1-10. α is set to 1, which is an optimal value derived from practical experience and suitable for the application. Simply increasing or decreasing this value will cause a performance penalty to the system. Assume that the problem x is being calculated here... j If the probability of belonging to class 1 is 1, then k is 1. (Question x) j The probability q of belonging to the category jk Let be a probability value between 0 and 1, and let's assume it's 0.70.

[0057] Finally, an auxiliary probability is used as the target probability distribution, and its auxiliary probability distribution is expressed as:

[0058]

[0059] in The sum of the probabilities of all questions representing the center of the k-th category is f. k′This represents the sum of probabilities of all questions at the k′-th category center. The objective function amplifies the original probability values ​​by performing operations such as squared sum normalization, enabling the algorithm to learn from high-confidence categories. After calculation by the objective function, this auxiliary probability value can be assumed to be 0.90.

[0060] Finally, the class probability distribution is brought closer to the target distribution using KL divergence, which yields q. jk The objective function gradually approaches the auxiliary probability of 0.90 from the original 0.70.

[0061]

[0062] The term C is used to distinguish contrast loss, indicating that the loss is a clustering loss;

[0063] Step 3-3: Utilize the joint optimization model of contrastive loss and clustering loss, specifically a loss optimization model for the general domain.

[0064]

[0065] Where η represents the weight of the contrast loss, used to balance the contrast loss and the clustering loss. η is set to 10. There is no easily predictable proportional relationship between this value and the performance. 10 is an optimal value suitable for the application, derived from practice. Simply increasing or decreasing this value will cause a performance loss in the system.

[0066] The beneficial effects achieved by this invention are as follows: By training a BERT pre-trained model on general domain sample data based on contrastive learning, the anisotropy problem in the semantic space of the BERT pre-trained model is solved; by clustering the questions in the knowledge base through knowledge base quality inspection, the noise problem of repeated and overlapping standard questions in the knowledge base is solved; by continuing to train the BERT general domain semantic representation model on business domain sample data based on contrastive clustering, the problem that the BERT general domain semantic representation model can only learn semantic information at the sample pair level and cannot learn high-level semantic information at the category level is solved.

[0067] The above embodiments are not intended to limit the present invention in any way. Any other improvements and applications made to the above embodiments by equivalent transformations shall fall within the protection scope of the present invention.

Claims

1. A semantic representation method for intelligent question answering that integrates contrastive learning and clustering, characterized in that: Step 1: Collect sample pairs of data from the general domain, load the open-source BERT pre-trained model, and use contrastive learning to train the BERT pre-trained model to obtain the general domain BERT semantic representation model. Step 2: Use clustering algorithms to perform quality checks on the knowledge base; Step 2 includes clustering the questions in the knowledge base based on similarity, identifying and filtering out non-overlapping standard questions, which includes the following specific steps: Step 2-1: Encode all questions in the knowledge base using the general domain BERT semantic representation model to obtain the sentence vector representation of each question; Step 2-2: Perform fast clustering using the cosine similarity of sentence vectors. When the cosine similarity of the sentence vectors of two questions is greater than a preset threshold, they are set as similar, and all sentence vectors are divided into more than one category. Steps 2-3: Detect whether all sentence vectors in each category belong to the same standard question. If any category of sentence vectors belongs to multiple standard questions and the proportion exceeds the preset ratio, then merge the questions under the category. Step 3: On the dataset of the business domain, use contrastive clustering to continue training the general domain BERT semantic representation model to obtain the business domain BERT semantic representation model; Step 3 includes sampling sample pairs from non-repeating standard queries and training a BERT semantic representation model for the business domain using a contrastive clustering joint loss function.

2. The intelligent question-answering semantic representation method integrating contrastive learning and clustering as described in claim 1, characterized in that, Step 1 includes the following specific steps: Step 1-1: Collect sample pairs of data from a general domain as training sample pairs; Steps 1-2: Build the BERT model and load the corresponding open-source pre-trained weights of the BERT model to obtain the BERT pre-trained model; Steps 1-3: Using the general domain dataset collected in Step 1-1, continue training the BERT pre-trained model with contrastive loss to obtain the general domain BERT semantic representation model.

3. The intelligent question-answering semantic representation method integrating contrastive learning and clustering as described in claim 2, characterized in that, In steps 1-2, the pre-trained weights are from the Roberta-WWM-Base model. In steps 1-3, the BERT pre-trained model is optimized using a contrastive loss function to obtain a general-domain BERT semantic representation model. The contrastive loss function is: Where I indicates that the loss is a comparative loss. The number of sentence pairs representing a batch. , These represent the index numbers of the batches to which the two groups of sentences belong, respectively. 1 and 2 represent the numbers of different sentences belonging to the same index. Represents the index Positive sample pairs constructed from sentences. Represents the index The sentence and the index The negative sample pairs constructed from the sentences. It is the temperature control coefficient of the control model.

4. The intelligent question-answering semantic representation method integrating contrastive learning and clustering as described in claim 3, characterized in that, In step 3, the weights of the MLP are used to represent each category. sentence vector Using the t-distribution to calculate the original problem Divided into the first The probabilities of each category are: in This indicates the number of standard questions in the knowledge base. express Category index, One of the categories in the index that represents the number of standard questions. This represents the sentence vector for each category. Indicate each question The sentence vector, Let represent the degrees of freedom of the t-distribution. Finally, an auxiliary probability is used as the target probability distribution, which is expressed as: in , representing the The sum of the probabilities of all problems centered on each category is then Representing the The sum of probabilities of all problems at each category center; the clustering loss is obtained by using KL divergence to bring the category probability distribution closer to the target distribution: The contrastive loss is used to distinguish between clustering losses; the joint loss is expressed as: in This indicates the weight of the comparative loss.

5. The intelligent question-answering semantic representation method integrating contrastive learning and clustering as described in claim 4, characterized in that, The number of sentence pairs is 256, and the temperature control coefficient is... The value is 0.

5. In step 3, The value is 10.

Images