A method and system for generating a semantic matching model based on remote supervision

Abstract

The application provides a generation method and system of a semantic matching model based on remote supervision, relates to the technical field of machine learning, and comprises the following steps: obtaining a plurality of original text data of a preset application scenario, and performing data enhancement on each original text data to obtain a plurality of enhanced text data, each enhanced text data being associated with a standard semantic category label corresponding to the preset application scenario; performing weight fine-tuning on a pre-trained language model according to each enhanced text data and the associated standard semantic category label to obtain a fine-tuned model; performing sentence embedding on each original text data according to the fine-tuned model to obtain a plurality of semantic vectors, and automatically labeling each semantic vector to obtain a plurality of semantic matching vector pairs and a plurality of semantic mismatching vector pairs; and training a semantic matching model according to each semantic matching vector pair and each semantic mismatching vector pair. The beneficial effect is that the cost of a large amount of manual labeling is effectively reduced, and the problem of cold start difficulty of the semantic matching model is relieved to a certain extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a method and system for generating a semantic matching model based on remote supervision. BACKGROUND

[0002] The most direct goal of semantic matching is to determine whether two sentences express the same meaning. The main tasks of sentence-oriented semantic matching are as follows: 1. finding a reasonable word embedding method to describe the key features; and 2. performing binary classification on the semantic similarity of the semantic vectors of two sentences.

[0003] The commonly used semantic matching technology in the industry generally realizes vector representation by word frequency dimension, word embedding of text, or sentence-level embedding using a pre-trained language model to calculate the similarity between sentences. However, only a threshold can be used to determine semantic matching based on the similarity, which has certain limitations. Another solution is to regard semantic matching as a binary classification task for sentence pairs, which can be based on word2vec+lstm or a pre-trained language model. However, a considerable amount of labeled semantic matching training data is required, and the training data with labels is often incomplete, which makes it difficult to start the model. Furthermore, to achieve a satisfactory accuracy, the amount of training data required is at least 200,000, which is too high for manual labeling and puts pressure on computing performance. SUMMARY

[0004] To solve the problems in the prior art, the present application provides a method for generating a semantic matching model based on remote supervision, comprising:

[0005] Step S1: obtaining a plurality of original text data of a predetermined application scenario, and performing data enhancement on each of the original text data to obtain a plurality of enhanced text data, each of the enhanced text data being associated with a standard semantic category label corresponding to the predetermined application scenario;

[0006] Step S2: performing weight fine-tuning on a pre-trained language model according to each of the enhanced text data and the associated standard semantic category label to obtain a fine-tuned model;

[0007] Step S3: performing sentence embedding on each of the original text data according to the fine-tuned model to obtain a plurality of semantic vectors, and automatically labeling each of the semantic vectors to obtain a plurality of semantic matching vector pairs and a plurality of semantic mismatching vector pairs;

[0008] Step S4: training a semantic matching model according to each of the semantic matching vector pairs and each of the semantic mismatching vector pairs.

[0009] Preferably, the step S1 comprises:

[0010] Step S11, obtaining each of the original text data from each of the business systems associated with the preset application scenario, each of the original text data being associated with a corresponding original semantic category label defined in the business system;

[0011] Step S12, standardizing each of the original semantic category labels into the standard semantic category label in the preset application scenario, and data fusing each of the original text data having the same standard semantic category label;

[0012] Step S13, performing category scoring on each of the standard semantic category labels according to each of the original text data after data fusion to obtain a corresponding score value, and determining whether the score value is greater than a preset score threshold:

[0013] If yes, adding each of the original text data associated with the corresponding standard semantic category label to a high-quality data set, and then turning to step S14;

[0014] If no, adding each of the original text data associated with the corresponding standard semantic category label to a low-quality data set, and then turning to step S15;

[0015] Step S14, performing word extraction on each of the original text data in the high-quality data set, and constructing a semantic dictionary based on each semantic representative word obtained by word extraction;

[0016] Step S15, configuring the standard semantic category label for each of the original text data in the low-quality data set according to the semantic dictionary;

[0017] Step S16, performing data enhancement on each of the original text data associated with each of the standard semantic category labels to obtain each of the enhanced text data.

[0018] Preferably, in the step S13, the score value is obtained by category scoring according to a data length and a data amount proportion of each of the original text data in the standard semantic category to all of the original text data.

[0019] Preferably, the step S14 comprises:

[0020] Step S141, respectively extracting keywords and topic words from each of the original text data in the high-quality data set;

[0021] Step S142, performing deduplication processing on the extracted keywords and topic words, and screening each of the semantic representative words from the deduplicated keywords and topic words.

[0022] Preferably, in the step S16, the data enhancement manner comprises:

[0023] performing non-keyword replacement on each of the original text data associated with each of the standard semantic category labels to form new text data, and taking each of the original text data and the new text data formed by the non-keyword replacement as the enhanced text data; and / or

[0024] performing selective random sampling on each of the original text data associated with each of the standard semantic category labels, and taking each of the original text data obtained by the random sampling as the enhanced text data associated with the standard semantic category label.

[0025] Preferably, the step S3 comprises:

[0026] Step S31, performing sentence embedding on each of the original text data according to the fine-tuned model to obtain a plurality of semantic vectors, and adding each of the semantic vectors to a vector set;

[0027] Step S32, performing text clustering on each of the semantic vectors in the vector set to obtain a plurality of cluster centers of clustering clusters formed after the text clustering, calculating the distance between the semantic vector as the cluster center and other semantic vectors for each clustering cluster, and determining whether the distance is within a preset threshold range:

[0028] If yes, the corresponding semantic vector is added to a fault set, and then the step S34 is turned to;

[0029] If no, the step S33 is turned to;

[0030] Step S33, determining whether the distance is not greater than a lower limit value of the threshold range:

[0031] If yes, the corresponding semantic vector is added to a trusted set, and then the step S34 is turned to;

[0032] If no, the corresponding semantic vector is added to an untrusted set, and then the step S34 is turned to;

[0033] Step S34, determining whether the trusted set is an empty set:

[0034] If yes, exit;

[0035] If not, each of the semantic vectors in the trusted set in the same cluster is automatically labeled as a plurality of semantic matching vector pairs between each of the semantic vectors in the trusted set in the same cluster, between each of the semantic vectors in the trusted set and each of the semantic vectors in the untrusted set of the same cluster, and between each of the semantic vectors in the trusted set and each of the semantic vectors in other clusters.

[0036] Step S35, the semantic vectors in each trusted set are removed from the vector set, and then return to step S32.

[0037] Preferably, the semantic matching model comprises a Bert model, an embedding layer of the Bert model is increased with an adversarial perturbation layer, an output of the Bert is connected with a softmax layer, an input of the Bert model is taken as an input of the semantic matching model, and an output of the softmax layer is taken as an output of the semantic matching model.

[0038] The application also provides a generation system of a semantic matching model based on remote supervision, which applies the above generation method, and comprises:

[0039] A data enhancement module is configured to obtain a plurality of original text data of a preset application scenario, and perform data enhancement on each of the original text data to obtain a plurality of enhanced text data, each of which is associated with a standard semantic category label corresponding to the preset application scenario.

[0040] A weight fine-tuning module is connected with the data enhancement module and configured to fine-tune a pre-trained language model according to each of the enhanced text data and the associated standard semantic category label to obtain a fine-tuned model.

[0041] A sample construction module is connected with the data enhancement module and the weight fine-tuning module respectively, and configured to perform sentence embedding on each of the original text data according to the fine-tuned model to obtain a plurality of semantic vectors, and automatically label each of the semantic vectors to obtain a plurality of semantic matching vector pairs and a plurality of semantic mismatching vector pairs.

[0042] A model training module is connected with the sample construction module and configured to train a semantic matching model according to each of the semantic matching vector pairs and each of the semantic mismatching vector pairs.

[0043] Preferably, the sample construction module comprises:

[0044] A semantic vector generation unit is configured to perform sentence embedding on each of the original text data according to the fine-tuned model to obtain a plurality of semantic vectors, and add each of the semantic vectors to a vector set.

[0045] a semantic vector clustering unit connected to the semantic vector generating unit, configured to perform text clustering on each semantic vector in the vector set to obtain a plurality of cluster centers of clustering formed after text clustering, for each cluster center, respectively calculate the distance between the semantic vector as the cluster center and other semantic vectors, and when the distance is within a preset threshold range, add the corresponding semantic vector to a fault set, and when the distance is not greater than a lower limit value of the threshold range, add the corresponding semantic vector to a trusted set, and when the distance is not within the threshold range and is greater than the lower limit value, add the corresponding semantic vector to an untrusted set;

[0046] a judgment unit connected to the semantic vector clustering unit, configured to automatically label each semantic vector in the trusted set in the same cluster as a plurality of semantic matching vector pairs when the trusted set is not an empty set, and automatically label each semantic vector in the trusted set of the same cluster and each semantic vector in the untrusted set, and each semantic vector in the trusted set and each semantic vector in other clusters as a plurality of semantic mismatching vector pairs;

[0047] an iteration unit connected to the judgment unit, configured to remove the semantic vectors in each trusted set from the vector set;

[0048] The semantic vector clustering unit iteratively performs text clustering on the removed vector set until the judgment unit determines that the trusted set is an empty set.

[0049] Preferably, the semantic matching model comprises a Bert model, an embedding layer of the Bert model is increased with an adversarial perturbation layer, an output of the Bert is connected with a softmax layer, an input of the Bert model is used as an input of the semantic matching model, and an output of the softmax layer is used as an output of the semantic matching model.

[0050] The above technical solution has the following advantages or beneficial effects:

[0051] 1) The automatic labeling of semantic matching vector pairs and semantic mismatching vector pairs is performed through remote supervision, which effectively reduces the cost of a large amount of manual labeling, and to some extent, alleviates the cold start difficulty of the semantic matching model;

[0052] 2) The pre-trained Bert model is used to realize the training of the semantic matching algorithm model in the form of a two-classification task of multiple sentences, and is assisted by an adversarial perturbation layer structure, so that the trained semantic matching model can better cope with the boundary division of semantic matching. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 For the preferred embodiments of the present application, a flowchart of a method for generating a semantic matching model based on remote supervision is provided.

[0054] Figure 2 For the preferred embodiments of the present application, a flowchart of a method for generating a semantic matching model based on remote supervision is provided.

[0055] Figure 3 For the preferred embodiments of the present application, a flowchart of a method for generating a semantic matching model based on remote supervision is provided.

[0056] Figure 4 For the preferred embodiments of the present application, a flowchart of a method for generating a semantic matching model based on remote supervision is provided.

[0057] Figure 5 For the preferred embodiments of the present application, a flowchart of a method for generating a semantic matching model based on remote supervision is provided. DETAILED DESCRIPTION

[0058] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. The present application is not limited to this embodiment, and other embodiments can also fall within the scope of the present application as long as they comply with the spirit of the present application.

[0059] In the preferred embodiments of the present application, based on the above-mentioned problems existing in the prior art, a method for generating a semantic matching model based on remote supervision is provided, as shown in Figure 1 The method comprises the following steps:

[0060] Step S1, obtaining a plurality of original text data of a preset application scenario, and performing data augmentation on each original text data to obtain a plurality of augmented text data, each augmented text data being associated with a standard semantic category label corresponding to the preset application scenario;

[0061] Step S2, performing weight fine-tuning on a pre-trained language model according to each augmented text data and the associated standard semantic category label to obtain a fine-tuned model;

[0062] Step S3, performing sentence embedding on each original text data according to the fine-tuned model to obtain a plurality of semantic vectors, and performing automatic labeling on each semantic vector to obtain a plurality of semantic matching vector pairs and a plurality of semantic mismatching vector pairs;

[0063] Step S4, training a semantic matching model according to each semantic matching vector pair and each semantic mismatching vector pair.

[0064] Specifically, the preset application scenarios include but are not limited to city operation scenarios, and the corresponding original text data includes but is not limited to 12345 citizen hotline data and city emergency event data. To solve the problem of excessive manual annotation cost of data set required for training of the semantic matching model, in the embodiment, the remote supervision is used to automatically annotate the semantic matching vector pair and the semantic mismatching vector pair. Before automatic annotation, since the data quality of the original text data directly obtained is relatively low, the data augmentation is first performed on the original text data, so as to prepare data for automatic annotation of remote supervision. The effect of data augmentation can determine whether the remote supervision process has the persistent low-loss ability.

[0065] Further specifically, the process of data augmentation is as shown in Figure 2 The step S1 includes:

[0066] In the step S11, each original text data is obtained from each business system associated with the preset application scenario, and each original text data is associated with an original semantic category label defined in the corresponding business system.

[0067] In the step S12, each original semantic category label is standardized as a standard semantic category label in the preset application scenario, and each original text data with the same standard semantic category label is fused.

[0068] In the step S13, each standard semantic category label is scored according to each original text data after data fusion to obtain a corresponding score value, and it is determined whether the score value is greater than a preset score threshold:

[0069] If yes, each original text data associated with the corresponding standard semantic category label is added to a high-quality data set, and then the step S14 is performed.

[0070] If no, each original text data associated with the corresponding standard semantic category label is added to a low-quality data set, and then the step S15 is performed.

[0071] In the step S14, each original text data in the high-quality data set is subjected to word extraction, and a semantic dictionary is constructed based on each semantic representative word obtained by word extraction.

[0072] In the step S15, each original text data in the low-quality data set is configured with a standard semantic category label according to the semantic dictionary.

[0073] In the step S16, each original text data associated with each standard semantic category label is subjected to data augmentation to obtain each augmented text data.

[0074] Specifically, in the embodiment, taking the preset application scenario as the city operation scenario as an example, the above business systems include but are not limited to the 12345 citizen hotline platform and the emergency event handling platform, and the corresponding original text data are respectively the 12345 citizen hotline data and the city emergency event data. Considering that the same type of event may be defined as the personnel injury category in the original semantic category label of the 12345 citizen hotline data and as the personnel casualty category in the original semantic category label of the city emergency event data, it is necessary to first fuse the categories of the data and expand the breadth of the semantic category, such as defining the standard semantic category label as the personnel injury category, which contains the above-mentioned personnel injury category and the personnel casualty category.

[0075] Further specifically, first, the category of each standard semantic category label is scored to obtain a corresponding score value. Preferably, the category is scored according to the data length and the data amount proportion of each original text data in all original text data to obtain a corresponding score value. It can be understood that the longer the data length of each original text data in the standard semantic category and the greater the data amount proportion, the higher the corresponding score value. The specific scoring rules can be customized according to the requirements. The higher the score value, the better the category quality of the corresponding original text data. The category quality can be quantified by pre-configuring a score threshold. Preferably, a score value greater than the score threshold is considered to be high-quality text data, otherwise it is low-quality text data. For high-quality original text data, reasonable and high semantic representative degree words are extracted as semantic representative words to construct a semantic dictionary, and then the standard semantic category label is configured for each original text data in the low-quality data set according to the semantic dictionary, so as to realize the labeling of events that are obviously misclassified or not classified by the business system, and enrich the semantic category.

[0076] The above process of extracting reasonable and high semantic representative degree words is as shown in Figure 3 The step S14 includes:

[0077] Step S141, respectively, key words and topic words are extracted from each original text data in the high-quality data set.

[0078] Step S142, the extracted key words and topic words are de-duplicated, and each semantic representative word is selected from the de-duplicated key words and topic words.

[0079] Specifically, in the embodiment, after the extracted key words and topic words are de-duplicated, unreasonable or expression manner inconsistent with the existing expression manner key words and topic words can be obtained by manual screening, and then each semantic representative word is obtained to further improve the effect of subsequent data enhancement.

[0080] In the process of the above category fusion, some data categories can be less or more distributed, resulting in extremely uneven event description category distribution after fusion, and therefore the quality and distribution of the category labels need to be further optimized through data enhancement means. Specifically, the data enhancement methods include:

[0081] non-keyword replacement is performed on each original text data associated with each standard semantic category label, and each original text data and the new text data formed by the non-keyword replacement are taken as enhanced text data; and / or

[0082] selective random sampling is performed on each original text data associated with each standard semantic category label, and each original text data obtained by the random sampling is taken as enhanced text data associated with the standard semantic category label.

[0083] Specifically, in the embodiment, for a standard semantic category with less data, the above non-keyword replacement can be used to increase the number of samples. The above non-keyword is a word that does not affect the core semantics, which can be an address entity or a time entity. For a standard semantic category with more data, the above random sampling method can be used to reduce the number of samples, so that the distribution of the number of events in each standard semantic category is uniform.

[0084] After the data enhancement is completed, the pre-trained language model can be fine-tuned to make the weight parameters of the pre-trained language model originally used for the full-scene scenario more suitable for the emergency event text classification task facing the urban operation scenario. The above pre-trained language model is preferably a Bert model. The remote supervision method of the application is to fine-tune the sentence embedding weight based on the bert event text business category classification task model by means of the existing relationship between the event text entities and the business category entities in the business system knowledge base, realize semantic vector representation, and then construct the semantic matching and non-matching relationship between two event text entities through the iterative clustering result of the semantic vector, that is, complete the annotation of the semantic matching data set.

[0085] Specifically, as shown in Figure 4 step S3 includes:

[0086] Step S31, according to the fine-tuned model, sentence embedding is performed on each original text data to obtain a plurality of semantic vectors, and each semantic vector is added to a vector set;

[0087] Step S32, text clustering is performed on each semantic vector in the vector set to obtain the centroid of a plurality of clustering clusters formed after text clustering, and the distance between the semantic vector as the centroid and other semantic vectors is calculated for each clustering cluster, and it is determined whether the distance is within a preset threshold range:

[0088] If yes, the corresponding semantic vector is added to the fault set, and then step S34 is turned to;

[0089] If no, step S33 is turned to;

[0090] In step S33, it is judged whether the distance is not greater than the lower limit value of the threshold range:

[0091] If yes, the corresponding semantic vector is added to the trusted set, and then step S34 is turned to;

[0092] If no, the corresponding semantic vector is added to the untrusted set, and then step S34 is turned to;

[0093] In step S34, it is judged whether the trusted set is an empty set:

[0094] If yes, the process is exited;

[0095] If no, each semantic vector pair between the semantic vectors in the trusted set in the same cluster is automatically labeled as a plurality of semantic matching vector pairs, each semantic vector in the trusted set in the same cluster and each semantic vector in the untrusted set, and each semantic vector in the trusted set and each semantic vector in other clusters are automatically labeled as a plurality of semantic non-matching vector pairs;

[0096] In step S35, the semantic vectors in each trusted set are removed from the vector set, and then step S32 is returned.

[0097] Specifically, in this embodiment, a weight oriented to the classification of emergency incident and 12345 hotline semantic scenarios can be obtained through weight fine-tuning, and the semantic vector obtained by performing sentence embedding on the original text data based on the weight has a relatively clear ability to divide the semantics of 12345 and emergency services in the semantic space. Then, the semantic vector generated by the word embedding is sent to a text clustering model based on cosine similarity, and the centroid of each cluster can be simply understood as the semantic center under each semantic cluster. By calculating the distance between each sample and the cluster center and setting the untrusted threshold and the trusted threshold, and manually recording the samples, the appropriate threshold range is found through repeated tests, the trusted set, the fault set and the untrusted set are divided, and finally the trusted samples in the trusted set (the sample distance from the cluster center is less than or equal to the lower limit value of the threshold range), the untrusted samples in the untrusted set (the sample distance from the cluster center is greater than or equal to the upper limit value of the threshold range), and the fault set. The semantic vectors between the samples in the trusted set in the cluster form a matching pair, the semantic vectors between the samples in the trusted set and the untrusted set in the same cluster do not form a matching pair, and the semantic vectors between the samples in the trusted set in the cluster and any sample in other clusters do not form a matching pair. Next, the trusted set obtained in the last iteration is removed and re-clustered, the new trusted set, untrusted set and fault set are obtained by calculating the cluster center. If there is no trusted sample in the trusted set, the iteration ends.

[0098] The above text is embedded by using the bert model weight fine-tuned for the text classification task, and due to the special attention mechanism of bert, the traditional word meaning limitation and structure limitation can be avoided, so that the word embedding vector can better express the context semantic features and business scenario semantic features. Then, the invention sends the semantic vector after word embedding into a text clustering model based on cosine similarity for iterative clustering. For each clustering result, the distance of each sample to the centroid is controlled to divide the trusted set, the untrusted set and the fault set.

[0099] For the trusted set, the untrusted set and the fault set, semantic matching pairs are assembled, and the granularity of semantic matching is controlled by controlling the radius of the trusted set, so as to realize the automatic labeling of the semantic matching dataset in the form of remote supervision.

[0100] Based on the automatic labeling of remote supervision of the technical solution, 2000 texts with rough business categories can be automatically labeled as 200-300 semantic matching datasets, and the data size will fluctuate within a small range with the granularity of semantic matching.

[0101] After the above clustering iteration is completed, the obtained semantic vector matching pairs and semantic vector non-matching pairs are the training set obtained by automatic labeling. Subsequently, the training of the semantic matching model can be performed based on the training set. In a preferred embodiment of the present application, the semantic matching model comprises a Bert model, an embedding layer of the Bert model is increased with an adversarial perturbation layer, an output of the Bert is connected with a softmax layer, an input of the Bert model is taken as an input of the semantic matching model, and an output of the softmax layer is taken as an output of the semantic matching model.

[0102] Specifically, in the present embodiment, Bert-wmm is selected as the language model in terms of the pre-training model. Since the BERT-base, Chinese published by Google officially does not consider the Chinese word segmentation in traditional NLP, the Bert-wmm applies the full word mask method to Chinese, uses the Chinese Wikipedia (including Simplified and Traditional Chinese) for training, and uses the Harbin Institute of Technology LTP as a word segmentation tool, that is, all Chinese characters constituting a word are masked. A softmax layer is added based on the Bert-wmm, and the purpose is to normalize the model output and convert the output into a probability that naturally has better interpretability. An adversarial perturbation model structure is added after the softmax layer to optimize the classification effect of the boundary problem of the small granularity. Semantic matching is essentially a multi-sentence binary classification downstream task based on Bert-wmm. By calling the pre-trained language model for the classification task and increasing the adversarial perturbation for fine-tuning, the training of the semantic matching model is realized.

[0103] The training of the semantic matching algorithm model is realized in the form of a multi-sentence binary classification task through the pre-trained Bert model, and the adversarial perturbation model structure is used to make the model better cope with the boundary division of semantic matching. The reason for adding the adversarial perturbation is that the assembly logic of the semantic matching pair under remote supervision cannot accurately distinguish the matching and non-matching in very subtle boundary problems, so the adversarial perturbation is added in the subsequent training process to optimize the semantic matching ability of the model for the boundary problem of the small granularity. The adversarial training is mainly performed by increasing the perturbation in the embedding layer of the Bert model. The output of the embedding layer is directly taken from the Embedding parameter matrix, so the Embedding parameter matrix can be directly perturbed. Although the diversity of the adversarial samples obtained in this way is less (because different samples share the same perturbation for the same token), it still plays a regularization role. The semantic matching model trained using the remote supervision labeled data set can achieve an accuracy of more than 95%. In business, it can accurately enable the analysis of 12345 hotline multi-party litigation and the analysis of the spatiotemporal semantic aggregation of the emergency.

[0104] In summary, in the city operation scenario, the semantic matching analysis is performed on the event description of the 12345 citizen hotline and the city emergency event in the city operation scenario, the automatic labeling of the target data set is realized by means of the external knowledge based on the pre-training language model and the deep learning technology and the remote supervision, and the intelligent analysis application of the upper layer facing the specific business is supported, such as the semantic merging in the time and space semantic dimension of the emergency event, the 12345 multi-person lawsuit and the one-person multi-lawsuit analysis and the analysis application based on the semantic matching algorithm model.

[0105] The application further provides a generation system of a semantic matching model based on remote supervision, which applies the generation method. Figure 5 As shown in the figure, the generation system comprises:

[0106] A data enhancement module 1 is configured to obtain a plurality of original text data of a preset application scenario, and perform data enhancement on each original text data to obtain a plurality of enhanced text data, each enhanced text data being associated with a standard semantic category label corresponding to the preset application scenario.

[0107] A weight fine-tuning module 2 is connected to the data enhancement module 1 and configured to fine-tune the pre-trained language model according to each enhanced text data and the associated standard semantic category label to obtain a fine-tuned model.

[0108] A sample construction module 3 is connected to the data enhancement module 1 and the weight fine-tuning module 2, and configured to perform sentence embedding on each original text data according to the fine-tuned model to obtain a plurality of semantic vectors, and automatically label each semantic vector to obtain a plurality of semantic matching vector pairs and a plurality of semantic non-matching vector pairs.

[0109] A model training module 4 is connected to the sample construction module 3 and configured to train a semantic matching model according to each semantic matching vector pair and each semantic non-matching vector pair.

[0110] In the preferred embodiment of the application, the sample construction module 3 comprises:

[0111] A semantic vector generation unit 31 is configured to perform sentence embedding on each original text data according to the fine-tuned model to obtain a plurality of semantic vectors, and add each semantic vector to a vector set.

[0112] The semantic vector clustering unit 32, connected to the semantic vector generation unit 31, is used for text clustering of each semantic vector in the vector set to obtain a plurality of cluster centers formed after text clustering, for each cluster center, respectively calculating the distance between the semantic vector as the cluster center and other semantic vectors, and when the distance is within a preset threshold range, adding the corresponding semantic vector to the fault set, and when the distance is not greater than the lower limit value of the threshold range, adding the corresponding semantic vector to the trusted set, and when the distance is not within the threshold range and greater than the lower limit value, adding the corresponding semantic vector to the untrusted set;

[0113] The judgment unit 33, connected to the semantic vector clustering unit 32, is used for automatically labeling each semantic vector in the trusted set in the same cluster as a plurality of semantic matching vector pairs when the trusted set is not an empty set, automatically labeling each semantic vector in the trusted set in the same cluster and each semantic vector in the untrusted set, and automatically labeling each semantic vector in the trusted set and each semantic vector in other clusters as a plurality of semantic mismatching vector pairs.

[0114] The iteration unit 34, connected to the judgment unit 33, is used for removing the semantic vectors in each trusted set from the vector set.

[0115] The semantic vector clustering unit 32 iteratively performs text clustering on the removed vector set until the judgment unit 33 judges that the trusted set is an empty set.

[0116] In the preferred embodiment of the present application, the semantic matching model includes a Bert model, the embedding layer of the Bert model is increased with an adversarial perturbation layer, the output of the Bert is connected with a softmax layer, the input of the Bert model is used as the input of the semantic matching model, and the output of the softmax layer is used as the output of the semantic matching model.

[0117] The above only describes the preferred embodiments of the present application, and does not limit the implementation and protection scope of the present application. For those skilled in the art, it should be realized that any equivalent replacement and obvious changes made according to the present application and the drawings should be included in the protection scope of the present application.

Claims

1. A method for generating a semantic matching model based on remote supervision, characterized in that, include: Step S1: Obtain several original text data of a preset application scenario, and perform data augmentation on each of the original text data to obtain multiple augmented text data. Each augmented text data is associated with a standard semantic category label corresponding to the preset application scenario. Step S2: Fine-tune the weights of the pre-trained language model based on the enhanced text data and the associated standard semantic category labels to obtain the fine-tuned model; Step S3: Based on the fine-tuned model, sentence embedding is performed on each of the original text data to obtain multiple semantic vectors, and each of the semantic vectors is automatically labeled to obtain multiple semantic matching vector pairs and multiple semantic mismatch vector pairs; Step S4: Train a semantic matching model based on each semantic matching vector pair and each semantic mismatch vector pair; Step S1 includes: Step S11: Obtain each of the original text data from the business systems associated with the preset application scenario, and each of the original text data is associated with a corresponding original semantic category label defined in the business system; Step S12: Standardize each of the original semantic category labels to the standard semantic category labels in the preset application scenario, and perform data fusion on each of the original text data with the same standard semantic category labels; Step S13: Based on the original text data after data fusion, each standard semantic category label is scored to obtain a corresponding score value, and it is determined whether the score value is greater than a preset score threshold. If so, the original text data associated with the corresponding standard semantic category label are added to the high-quality data set, and then the process proceeds to step S14; If not, the original text data associated with the corresponding standard semantic category label are added to the low-quality data set, and then the process proceeds to step S15; Step S14: Extract words from each of the original text data in the high-quality data set, and construct a semantic dictionary based on the semantic representative words obtained from the word extraction; Step S15: Configure the standard semantic category label for each of the original text data in the low-quality data set according to the semantic dictionary; Step S16: Perform data augmentation on the original text data associated with each of the standard semantic category labels to obtain the augmented text data.

2. The generation method according to claim 1, characterized in that, In step S13, the corresponding score value is obtained by classifying the original text data in each standard semantic category according to the proportion of the data length and data volume of each original text data to all the original text data.

3. The generation method according to claim 1, characterized in that, Step S14 includes: Step S141: Extract keywords and topic terms from each of the original text data in the high-quality dataset; Step S142: Deduplication is performed on the extracted keywords and subject terms, and semantic representative words are obtained by filtering from the deduplicated keywords and subject terms.

4. The generation method according to claim 1, characterized in that, In step S16, the data augmentation methods include: Non-keyword replacements are performed on the original text data associated with each of the aforementioned standard semantic category labels, so that the original text data and the new text data formed by the non-keyword replacements are used as the enhanced text data; and / or Selectively random sample each of the original text data associated with each of the standard semantic category labels, and use the original text data corresponding to each of the randomly sampled original text data as the enhanced text data associated with the standard semantic category labels.

5. The generation method according to claim 1, characterized in that, Step S3 includes: Step S31: Based on the fine-tuned model, perform sentence embedding on each of the original text data to obtain multiple semantic vectors, and add each of the semantic vectors to the vector set; Step S32: Perform text clustering on each semantic vector in the vector set to obtain the centroids of multiple clusters formed after text clustering. For each cluster, calculate the distance between the semantic vector serving as the centroid and other semantic vectors, and determine whether the distance is within a preset threshold range. If so, the corresponding semantic vector is added to the tomography set, and then proceed to step S34; If not, proceed to step S33; Step S33: Determine whether the distance is not greater than the lower limit of the threshold range. If so, the corresponding semantic vector is added to the trusted set, and then the process proceeds to step S34; If not, the corresponding semantic vector is added to the untrusted set, and then proceeds to step S34; Step S34: Determine whether the trusted set is an empty set. If so, then exit; If not, then each semantic vector in the trusted set within the same cluster will be automatically labeled as multiple semantic matching vector pairs, and each semantic vector in the trusted set of the same cluster will be automatically labeled as multiple semantic mismatch vector pairs with each semantic vector in the untrusted set, and each semantic vector in the trusted set will be automatically labeled as multiple semantic mismatch vector pairs with each semantic vector in other clusters. Step S35: Remove the semantic vector from each of the trusted sets from the vector set, and then return to step S32.

6. The generation method according to claim 1, characterized in that, The semantic matching model includes a BERT model, the embedding layer of which adds an adversarial perturbation layer, the output of which is connected to a softmax layer, the input of which is used as the input of the semantic matching model, and the output of which is used as the output of the semantic matching model.

7. A generative system for a semantic matching model based on remote supervision, characterized in that, The generation system, employing the generation method as described in any one of claims 1-6, comprises: The data augmentation module is used to acquire several original text data of a preset application scenario, and perform data augmentation on each of the original text data to obtain multiple augmented text data. Each of the augmented text data is associated with a standard semantic category label corresponding to the preset application scenario. The weight fine-tuning module, connected to the data augmentation module, is used to fine-tune the weights of the pre-trained language model based on each augmented text data and the associated standard semantic category label to obtain the fine-tuned model. The sample construction module is connected to the data augmentation module and the weight fine-tuning module respectively. It is used to embed sentences into each of the original text data according to the fine-tuned model to obtain multiple semantic vectors, and to automatically label each of the semantic vectors to obtain multiple semantic matching vector pairs and multiple semantic mismatch vector pairs. The model training module, connected to the sample construction module, is used to train a semantic matching model based on each semantic matching vector pair and each semantic mismatch vector pair.

8. The generation system according to claim 7, characterized in that, The sample construction module includes: A semantic vector generation unit is used to perform sentence embedding on each of the original text data according to the fine-tuned model to obtain multiple semantic vectors, and to add each of the semantic vectors into a vector set; A semantic vector clustering unit, connected to the semantic vector generation unit, is used to perform text clustering on each semantic vector in the vector set to obtain the centroids of multiple clusters formed after text clustering. For each cluster, the distance between the semantic vector serving as the centroid and other semantic vectors is calculated. When the distance is within a preset threshold range, the corresponding semantic vector is added to the tortuosity set. When the distance is not greater than the lower limit of the threshold range, the corresponding semantic vector is added to the trust set. When the distance is not within the threshold range and is greater than the lower limit, the corresponding semantic vector is added to the untrust set. The judgment unit, connected to the semantic vector clustering unit, is used to automatically label the semantic vectors in the trusted set within the same cluster as multiple semantic matching vector pairs when the trusted set is not an empty set, and to automatically label the semantic vectors in the trusted set within the same cluster with the semantic vectors in the untrusted set, as well as the semantic vectors in the trusted set with the semantic vectors in other clusters as multiple semantic mismatch vector pairs; An iteration unit, connected to the judgment unit, is used to remove the semantic vector from each of the trust sets from the vector set; The semantic vector clustering unit iteratively performs text clustering on the removed vector set until the judgment unit determines that the trustworthy set is an empty set.

9. The generation system according to claim 7, characterized in that, The semantic matching model includes a BERT model, the embedding layer of which adds an adversarial perturbation layer, the output of which is connected to a softmax layer, the input of which is used as the input of the semantic matching model, and the output of which is used as the output of the semantic matching model.

Images