Method, device and system for obtaining training data, and storage medium

Abstract

Some embodiments of the present application provide a method, device, system and storage medium for obtaining training data, the method comprising: obtaining semantic representation vectors of each sentence in a plurality of sentences according to a target semantic representation model; obtaining similarity values between any sentence and the remaining sentences in the plurality of sentences according to the semantic representation vectors and a similarity algorithm, to obtain a plurality of similarity values; if it is confirmed that the any sentence is similar to any reference sentence and the any sentence and the any reference sentence do not belong to the same sentence cluster according to the size relationship of the plurality of similarity values, then it is confirmed that the any sentence and the reference sentence form a set of negative sample data. Some embodiments of the present application can construct negative sample data with semantic matching level, and thus the text matching model trained by using the negative sample data has strong semantic matching ability.

Description

Technical Field

[0001] This application relates to the field of text matching technology. Specifically, embodiments of this application relate to methods, apparatus, systems, and storage media for acquiring training data. Background Technology

[0002] When training a text matching model, it is necessary to construct negative samples for the model, and the quality of the constructed negative samples determines the matching effect of the trained text matching model.

[0003] The following example uses the question matching model in text matching to illustrate the current problems in constructing negative sample data.

[0004] Question matching models take a pair of questions as input and output whether the question pair is similar. Common question-answering systems calculate similarity by matching a user-input question with all questions in the knowledge base. Compared to models with single text input, such as news classification models, question matching models require constructing positive and negative samples during data preparation. The inventors of this application discovered that because related technologies only confirm whether two questions match at the literal or lexical level when constructing negative samples, without verifying semantic matching, the question matching model trained using this negative sample data lacks semantic matching ability between sentences.

[0005] Therefore, how to improve the quality of negative sample training data has become an urgent technical problem to be solved. Summary of the Invention

[0006] The purpose of this application is to provide a method, apparatus, system and storage medium for acquiring training data. The technical solutions of the embodiments of this application can construct a semantic-level negative sample dataset, thereby enabling the text matching model to be trained to have a strong semantic matching ability.

[0007] In a first aspect, embodiments of this application provide a method for acquiring training data. The method includes: acquiring semantic representation vectors of each statement in a plurality of statements according to a target semantic representation model; acquiring similarity values ​​between any statement in the plurality of statements and the remaining statements according to the semantic representation vectors and a similarity algorithm, thereby obtaining a plurality of similarity values, wherein the remaining statements constitute a reference statement for any statement; if it is confirmed that any statement is similar to any reference statement based on the magnitude relationship of the plurality of similarity values, and it is confirmed that any statement and any reference statement do not belong to the same statement cluster, then it is confirmed that any statement and the reference statement constitute a set of negative sample data.

[0008] This application embodiment applies the target semantic representation model to the acquisition of negative sample data. Since the semantic representation vector obtained by the semantic representation model can express the semantic features of each sentence, the similar sentences obtained based on these vectors have semantic matching attributes, which improves the quality of negative sample data and ultimately improves the training results of the sentence matching model trained based on the negative sample.

[0009] In some embodiments, obtaining the similarity value between any statement among the plurality of statements and the remaining statements based on the semantic representation vector and the similarity algorithm includes: forming a statement pair by combining any statement among the plurality of statements with the remaining statements to obtain a plurality of statement pairs, wherein any statement pair among the plurality of statement pairs includes the stated statement and a reference statement of the stated statement; reading two semantic representation vectors corresponding to any statement pair among the plurality of statement pairs from a storage unit, wherein the storage unit is used to store the semantic representation vectors of each statement among the plurality of statements; and obtaining the similarity value of any statement pair based on the two semantic representation vectors and the similarity algorithm.

[0010] This application embodiment reads paired semantic representation vectors by forming sentence pairs, and then calculates the similarity value of each sentence pair based on the paired semantic representation vectors. By forming multiple sentence pairs, it is convenient to read a pair of semantic representation vectors from the storage unit one by one and calculate the similarity between the two sentences to confirm whether the two sentences are similar.

[0011] In some embodiments, determining that any statement is similar to any reference statement based on the magnitude relationship of the plurality of similarity values ​​includes: sorting the plurality of similarity values ​​according to a preset rule to obtain a sorting queue; and determining that any statement is similar to any reference statement based on the position of the similarity value corresponding to any reference statement in the sorting queue.

[0012] This application embodiment can quickly determine the relative position of the similarity value between any statement and any reference statement in the ranking of all similarity values ​​by sorting, thereby making the similarity probability between the selected reference statements and the statements to be analyzed greater and improving the quality of the selected negative sample data.

[0013] In some embodiments, sorting the plurality of similarity values ​​according to a preset rule includes: sorting the similarity values ​​in descending order; confirming that any statement is similar to any reference statement based on the position of the similarity value corresponding to any reference statement in the sorting queue includes: if it is confirmed that the similarity value corresponding to any reference statement is located in the sorting queue before the set position, then it is confirmed that any statement is similar to the reference statement.

[0014] This application embodiment sorts the similarity values ​​in descending order to obtain the sentence pairs with the highest similarity values, confirming that any sentence is similar to the reference sentence, which facilitates the rapid screening of negative sample data.

[0015] In some embodiments, sorting the plurality of similarity values ​​according to a preset rule includes: sorting the similarity values ​​in ascending order; confirming that any statement is similar to any reference statement based on the position of the similarity value corresponding to any reference statement in the sorting queue includes: if it is confirmed that the similarity value corresponding to any reference statement is located after a set position in the sorting queue, then it is confirmed that any statement is similar to the reference statement.

[0016] This application embodiment sorts the similarity values ​​in ascending order to obtain the sentence pairs with the lower similarity values, confirming that any sentence is similar to the reference sentence. This facilitates the rapid acquisition of negative sample data and provides another possible implementation example compared to the sorting method from largest to smallest.

[0017] In some embodiments, before confirming that any statement does not belong to the same statement cluster as any reference statement, the acquisition method further includes: acquiring all similar statements of each standard statement according to a clustering algorithm to obtain a statement cluster corresponding to each standard statement; or, clustering all statements into multiple statement clusters according to a clustering algorithm.

[0018] This application embodiment pre-divides the statement clusters corresponding to each standard statement by using a clustering algorithm, that is, counts all similar statements for each statement, or divides all statements into multiple clusters, which facilitates subsequent confirmation by combining similarity values ​​whether any statement and the reference statement are a set of negative sample data, thus improving the efficiency of the acquisition method.

[0019] In some embodiments, the similarity algorithm employs a cosine similarity algorithm.

[0020] This application embodiment uses a cosine similarity algorithm to easily calculate the similarity between two statements in a statement pair, thereby improving the overall efficiency of the method's implementation.

[0021] In some embodiments, before obtaining the semantic representation vectors of each statement in multiple statements according to the target semantic representation model, the acquisition method further includes: preprocessing the acquired original statements to obtain preprocessed data, wherein the preprocessing includes removing noise from the original statements and / or segmenting statements in the original statements whose length is greater than a set threshold; dividing the preprocessed data into a training dataset and a validation dataset; training the constructed initial semantic representation model according to the data in the training dataset to obtain a semantic representation model to be validated; and confirming that the semantic representation model to be validated passes validation according to the validation dataset to obtain the target semantic representation model.

[0022] This application embodiment obtains a target semantic representation model by training and validating a semantic representation model. This target semantic representation model can obtain the representation vector of each sentence, has a high accuracy in semantic matching, and has strong semantic matching capabilities.

[0023] In some embodiments, the similarity algorithm includes any one of the cosine similarity algorithm, Jaccard similarity algorithm, and Pearson correlation algorithm.

[0024] This application provides a variety of similarity calculation algorithms, offering a wide range of choices, and allowing for the selection of different algorithms for different application scenarios.

[0025] Secondly, embodiments of this application provide a method for training a text matching model, which trains the text matching model using negative sample data obtained by the method in any embodiment of the first aspect.

[0026] This application provides a method for training a text matching model using negative sample data obtained by any of the methods in the first aspect described above, so that the trained text matching model has strong semantic matching ability and high accuracy.

[0027] Thirdly, embodiments of this application provide a training data acquisition device, comprising: an acquisition module configured to acquire semantic representation vectors of each statement in a plurality of statements according to a target semantic representation model; a calculation module configured to acquire similarity values ​​between any statement in the plurality of statements and the remaining statements according to the semantic representation vectors and a similarity algorithm, thereby obtaining a plurality of similarity values, wherein the remaining statements constitute a reference statement for any statement; and a data acquisition module configured to, if the relationship between the plurality of similarity values ​​confirms that any statement is similar to any reference statement, and confirms that any statement and any reference statement do not belong to the same statement cluster, then confirm that any statement and the reference statement constitute a set of negative sample data.

[0028] Fourthly, embodiments of this application provide a training apparatus for a text matching model, the apparatus comprising one or more modules, the one or more modules being configured to train the text matching model using negative sample data obtained by the method of any embodiment in the first aspect.

[0029] Fifthly, embodiments of this application provide a system comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers perform operations of the corresponding methods of any embodiment in the first aspect.

[0030] In a sixth aspect, embodiments of this application provide one or more computer storage media for storing instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of a corresponding method of any embodiment in the first aspect. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a method for training a semantic representation model to obtain a target semantic representation model, provided in an embodiment of this application;

[0033] Figure 2 A flowchart illustrating the method for acquiring training data provided in this application embodiment;

[0034] Figure 3 A flowchart illustrating the training method based on the pre-trained language model BERT provided in this application embodiment;

[0035] Figure 4 A flowchart illustrating the method for obtaining training data for a question matching model based on a target BERT model, as provided in this application embodiment;

[0036] Figure 5 A block diagram illustrating the composition of the training data acquisition device provided in the embodiments of this application. Detailed Implementation

[0037] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0038] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0039] In related technical examples, question matching models belong to the category of text matching models. A question matching model takes a pair of questions as input and outputs whether the question pair is similar. A common application scenario is in question answering (QA) systems, where user-input questions are matched with questions in the system's knowledge database to calculate similar questions and thus obtain the corresponding answers. Because question matching models take a pair of questions as input, compared to single-text input models such as news classification models, they require constructing positive and negative samples during the data preparation phase. Furthermore, industry research has shown that the quality of negative samples often determines the upper limit of the model's performance. In practical applications, the labeled datasets collected when training question matching models are typically in the form of standard questions and similar questions. Positive samples are constructed by combining two similar questions under the same standard question. Negative samples are typically constructed based on literal, lexical-level matching relationships, using random sampling methods to obtain negative samples. As described in the background section, this method of constructing negative samples only considers the matching problem at the lexical level, which results in the question matching model trained on this data lacking linguistic knowledge and semantic matching capabilities.

[0040] As can be seen from the aforementioned technologies, the construction of negative sample data in traditional question matching models is often based solely on random sampling methods. This method produces overly simplistic negative samples, leading to overfitting during model training. There are also negative sample construction methods that employ simple unsupervised similarity algorithms, but these methods only consider literal or lexical matching relationships and lack semantic-level matching capabilities.

[0041] In view of this, the embodiments of this application apply the semantic representation model to the method of obtaining negative sample data of the text matching model to be trained. This method achieves the effect of negative sample data enhancement, reduces the probability of overfitting in the training model stage, and the obtained negative sample data is closer to the negative sample, so that the text matching model trained using these negative sample data also has a strong semantic matching ability.

[0042] For example, in some embodiments of this application, the Bidirectional Encoder Representations from Transformers (BERT) model is selected as the semantic model for extracting semantic representation vectors of sentences. It is understood that BERT is a pre-trained language model, a cutting-edge field in Natural Language Processing (NLP), deeply applied in various NLP tasks such as text classification, text matching, and entity recognition. Since Google released the open-source pre-trained BERT model in 2019, research and applications in this field have become increasingly active. The essence of pre-training is that model parameters are no longer randomly initialized, but trained through specific tasks. For example, the Masked Language Model is pre-trained on a large-scale text corpus and then fine-tuned on a small dataset for a specific task. This approach has become a new paradigm for training NLP models, with the advantage of reducing the cost and training difficulty of models for specific NLP tasks and improving model performance.

[0043] It is understood that some embodiments of this application require the semantic representation model to be trained in advance to obtain a target semantic representation model in order to represent each statement through semantic representation vectors. The target semantic representation model can extract the semantic features of each input statement and obtain the semantic representation vector of each statement.

[0044] The following example illustrates the process of training a semantic representation model to obtain a target semantic representation model.

[0045] like Figure 1 As shown, some embodiments of this application provide a method for training a semantic representation model to obtain a target semantic representation model. The method includes: S210, preprocessing the acquired original sentences to obtain preprocessed data, wherein the preprocessing includes removing noise from the original sentences and / or segmenting sentences in the original sentences whose length exceeds a set threshold. S220, dividing the preprocessed data into a training dataset and a validation dataset. S230, training the constructed initial semantic representation model based on the data in the training dataset to obtain a semantic representation model to be validated. S240, confirming that the semantic representation model to be validated passes validation based on the validation dataset, thus obtaining the target semantic representation model.

[0046] It should be noted that raw statements need to be collected before executing S210. For example, in some embodiments of this application, statements labeled or related to the business QA system are collected. These statements are not limited to text data such as standard questions, similar questions, recommended questions, and standard answers.

[0047] To improve the quality of data in the training and validation sets, in some embodiments of this application, S210 includes removing noise from the original statement (e.g., special characters, spaces, garbled characters). Specifically, the noise in the original statement can be cleaned up using a constructed regular expression.

[0048] To improve the quality of data in the training and validation sets, in some embodiments of this application, S210 also includes truncation of excessively long statements, for example, truncation of excessively long dialogue text using a Python script.

[0049] It is understood that in some embodiments of this application, the collected raw statements may also be both denoised and truncated.

[0050] To improve the accuracy of semantic features extracted by the obtained target semantic representation model, it is necessary to reasonably divide the training set and validation set. For example, in some embodiments of this application, Python can be used to split the preprocessed dataset into training and validation sets according to an appropriate ratio. The training set is used to train the semantic representation model, and the validation set verifies whether the semantic representation model obtained after training from the training set data meets the requirements of the target semantic representation model.

[0051] After obtaining the target semantic representation model, the file corresponding to the trained target semantic representation model can be saved.

[0052] It should be noted that the semantic representation models of some embodiments of this application include any one of the following: the BERT model, the text classification convolutional neural network model (i.e., the TextCNN model), and the Transformer model.

[0053] The following section uses the trained target semantic representation model as an example to illustrate the specific process of obtaining training data provided in the embodiments of this application.

[0054] Please see Figure 2 , Figure 2 A flowchart illustrating the method for acquiring training data provided in this application embodiment.

[0055] The training data acquisition method provided in this application embodiment may include: S110, acquiring the semantic representation vector of each statement in a plurality of statements according to the target semantic representation model; S120, acquiring the similarity value between any statement in the plurality of statements and the remaining statements according to the semantic representation vector and a similarity algorithm, thereby obtaining a plurality of similarity values, wherein the remaining statements constitute a reference statement for any statement; S130, if it is confirmed that any statement is similar to any reference statement based on the magnitude relationship of the plurality of similarity values, and it is confirmed that any statement and any reference statement do not belong to the same statement cluster, then it is confirmed that any statement and the reference statement constitute a set of negative sample data.

[0056] The above process is illustrated below by example.

[0057] The target semantic representation model involved in S110 is based on the above... Figure 1 The method used to train the data is different from the method used in some embodiments of this application. Figure 1 The training process yields the target semantic representation model. Figure 1 The training process is only used as a specific example.

[0058] It is understandable that multiple statements need to be collected in advance before executing S110. For example, in some embodiments of this application, the multiple statements in S110 are a dataset obtained by collecting manually annotated standard questions and similar questions corresponding to the standard questions from a relevant business question-and-answer system.

[0059] S110 includes, for example, inputting each of the multiple statements into a target semantic representation model, which outputs a semantic representation vector corresponding to each statement. This is because the trained target semantic representation model can extract the semantic representation vector of the input statement.

[0060] In some embodiments of this application, S110 further includes storing each semantic representation vector collected by the target semantic representation model so that the similarity between pairs of statements can be calculated later according to the similarity algorithm.

[0061] To obtain the similarity value between each statement and the remaining statements, it is necessary to calculate the similarity between the statement and the remaining statements. Therefore, in some embodiments of this application, S120 includes: forming a statement pair with any statement from the plurality of statements obtained in S110 and the remaining statements respectively, thereby obtaining a plurality of statement pairs, wherein any statement pair includes the statement and a reference statement; reading two semantic representation vectors corresponding to any statement pair from the storage unit, wherein the storage unit is used to store the semantic representation vectors of each statement in the plurality of statements; and obtaining the similarity value of the statement pair based on the two semantic representation vectors and a similarity algorithm.

[0062] For example, S110 yields 10 statements. Taking the first statement among these ten statements as an example, S120 is illustrated. Executing S120 involves pairing the first statement with each of the remaining nine statements to obtain nine statement pairs. Then, the similarity value of each of these nine statement pairs is calculated, thus obtaining the similarity values ​​corresponding to the first statement.

[0063] For example, in some embodiments of this application, the semantic representation vectors of each statement obtained in S110 are stored in a storage unit. Using the data combination function in any machine language (i.e., Python, Java, etc.), any statement among the multiple statements is combined with the remaining statements to form a binary combination, i.e., a statement pair, resulting in multiple statement pairs. The two semantic representation vectors corresponding to any statement pair are obtained from the storage unit, and the similarity of the statement pair is calculated using a similarity algorithm.

[0064] To obtain reference statements that form negative sample data with any given statement, it is necessary to filter out statements that have high semantic similarity to the given statement but do not belong to the same statement cluster as the reference statement (i.e., statements that were not assigned to a statement cluster by the clustering algorithm or were manually labeled as dissimilar) and use them to form negative sample data with the given statement. For example, in some embodiments of this application, the process of determining the similarity between any given statement and any reference statement based on the magnitude relationship of the multiple similarity values ​​in S130 includes: sorting all the obtained similarity values ​​corresponding to the given statement according to a preset rule to obtain a sorting queue; and determining the similarity between the given statement and the reference statement based on the position of the similarity value corresponding to the reference statement in the sorting queue.

[0065] It should be noted that the statement cluster is obtained by using a clustering algorithm to obtain all similar statements of each standard statement, thus obtaining the statement cluster corresponding to each standard statement; or, all statements are clustered into multiple statement clusters using a clustering algorithm.

[0066] For example, in some embodiments of this application, multiple standard statements are first obtained within the business system. Then, a clustering algorithm is used to group similar statements among the standard statements in the system into a statement cluster, thereby obtaining the statement cluster corresponding to each standard statement. Finally, multiple statement clusters are obtained. The standard questions are obtained manually or through other means (e.g., machine processing), and each standard question can represent the specific semantics of the corresponding cluster.

[0067] In some other embodiments of this application, multiple statements are first collected in the business system, and then a clustering algorithm is used to divide all the collected statements into multiple statement clusters.

[0068] In addition, clustering algorithms include, but are not limited to, K-means clustering, mean-drift clustering, Gaussian mixture model expectation-maximum clustering, and agglomerative hierarchical clustering.

[0069] In order to identify a reference statement that is semantically similar to any statement through a similarity value, in some embodiments of this application, the similarity algorithm in S120 can be a cosine similarity algorithm. Then S130 includes: sorting the similarity values ​​in descending order; if it is confirmed that the similarity value corresponding to any reference statement is located in the sorting queue before the set position, then it is confirmed that any statement is similar to the reference statement.

[0070] It should be noted that the similarity algorithms used above include, but are not limited to, cosine similarity algorithm, Jaccard similarity algorithm, and Pearson correlation algorithm.

[0071] In some embodiments of this application, negative sample data for any statement is obtained by filtering reference statements that are not in the same statement family as the statement. For example, if any statement is a first question, S130 includes: obtaining a negative sample data setting value (e.g., the value is 5, meaning 5 pairs of negative sample data are found for the statement). The similarity values ​​obtained in S120 are arranged in descending order. The sorting results are traversed from largest to smallest. If it is confirmed that a certain reference statement does not belong to the same question family as the first question, and the similarity value corresponding to the reference statement is ranked higher, then the reference statement can form a pair of negative sample data with the first question. This process is repeated to find the same number of reference statements as the number of negative sample data setting values, that is, all negative sample data for the first question are found.

[0072] It is understandable that the similarity values ​​can be sorted from smallest to largest, and the reference statements at the bottom of the sort can be used as similar statements to the first question. To avoid repetition, this will not be elaborated on further.

[0073] The following section uses BERT, a semantic representation model, as an example, and combines this with the process of obtaining negative sample data for a question matching model to illustrate the training process of the semantic representation model and the process of obtaining negative sample data.

[0074] See appendix Figure 3 , Figure 3 The flowchart of a training method based on the pre-trained language model BERT provided in some embodiments of this application is shown. The training method includes:

[0075] S101, Data Acquisition.

[0076] Collecting raw data of relevant texts from the business QA system may include collecting labeled or related text data from the business QA system, including but not limited to standard questions, similar questions, recommended questions, and standard answers.

[0077] S102, Data Preprocessing.

[0078] Since the raw data collected by S101 contains a lot of noise, such as meaningless special characters, spaces, and garbled characters, S102 can use regular expressions to clean up this noise.

[0079] If the length of the dialogue text in the raw data collected by S101 exceeds the set threshold, S102 will use a Python script with segmentation function to truncate it.

[0080] S103, Dataset Preparation.

[0081] S103 divides the raw data processed by S102 into a training dataset and a validation dataset according to a set ratio, which are used for model training and validation, respectively.

[0082] S104, training model.

[0083] The language pre-trained model BERT is trained using the training dataset obtained in S103 to obtain the BERT model to be validated. Then, the validation dataset is used to confirm that the BERT model to be validated passes the validation, thus obtaining the target BERT model. The target BERT model contains the target model file.

[0084] See appendix Figure 4 , Figure 4 The flowchart of the method for obtaining training data for a question matching model based on a target BERT model provided in this application embodiment is shown. The specific steps of the method for obtaining training data include:

[0085] S201, Collect question cluster data, which includes the following sub-steps:

[0086] Clustering algorithms are used to obtain all similar statements of the standard question, and similar questions corresponding to the same standard question are marked as a question cluster. Multiple standard questions correspond to multiple question clusters.

[0087] As an example, the QA system is an insurance claims system. Standard questions in this system include: "When will the claim payment arrive?". Using the K-means clustering algorithm, similar questions to this standard question include: "How many business days does it usually take for the payment to arrive?", "How many days does it usually take to successfully process a claim?", "How long does it usually take to process a claim after reporting it?", "How long does the claim processing usually take?", "Why hasn't my claim been processed yet?", "Why hasn't my payment arrived?", "When will the claim be processed?", "I've already applied for a claim, how long will it take to arrive?", etc. After executing S201, the claim payment time and all similar questions will be manually labeled into a question cluster.

[0088] As another example, clustering algorithms can be used to cluster all statements into multiple statement clusters. The QA system is an insurance claims system. All questions in this system include: Claim settlement time. How many business days does it usually take for the money to arrive? How many days does it usually take to successfully settle a claim? How long does it usually take to process a claim after reporting an incident? How to change the beneficiary? Hello, I want to change the beneficiary. Modify beneficiary. Change insurance beneficiary. Hello, can I change my beneficiary? How do I change the beneficiary? etc. After executing S201, the clustering algorithm will cluster all statements into two statement clusters. One statement cluster is about the claim settlement time, and the other is about changing the beneficiary.

[0089] S202, obtain the question representation vector based on the target BERT model (as a specific instance of the target semantic representation model).

[0090] Using the BERT model Figure 3 The target BERT model is trained using the following process: each question is input into the target BERT model, which outputs semantic representation vectors for each question in multiple question clusters, and stores all semantic representation vectors in a storage unit. For example, the storage unit may include external memory or a disk.

[0091] S203, calculate the cosine similarity of question pairs.

[0092] Each question is combined with the remaining questions individually (for example, using a Python script with combination capabilities) to obtain multiple question pairs (a specific example of a statement pair). Then, two semantic representation vectors corresponding to each question pair are retrieved from the storage unit in S202. Finally, based on the two semantic representation vectors and a similarity algorithm, the similarity value of any question pair is obtained.

[0093] S204, sort and extract negative samples to form the training dataset.

[0094] The steps are illustrated below using the first and second questions as examples.

[0095] Sort all similarity values ​​obtained in S203 corresponding to the first question in descending order. Iterate through the sorted results in ascending order. If the similarity value of the second question and the first question form a question pair that ranks higher and is confirmed to be outside the same question cluster, then the first and second questions constitute negative sample data. When the number of negative samples found exceeds a set value, all negative sample data is obtained. This negative sample data can be used as training data for the similarity statement recognition model to be trained.

[0096] It is understandable that the similar sentence recognition model to be trained is a type of text matching model.

[0097] In addition, this application embodiment also provides a method for training a text matching model, which uses negative sample data obtained by the above-described method for obtaining training data to train the text matching model.

[0098] Unlike related technologies that only consider the similarity of characters or words between sentences when acquiring negative sample data, the embodiments of this application apply the semantic features of each sentence extracted by the target semantic representation model to the acquisition of training data for the text matching model, thereby improving the semantic matching ability of the model.

[0099] Please refer to Figure 5 , Figure 5 A block diagram of a training data acquisition apparatus provided in an embodiment of this application is shown. It should be understood that this training data acquisition apparatus is similar to the one described above. Figure 2 Corresponding to the method embodiments, it is able to perform each step involved in the above method embodiments. The specific functions of the training data acquisition device can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here.

[0100] Figure 5 The training data acquisition device includes at least one software function module that can be stored in a memory or embedded in the training data acquisition device in the form of software or firmware. The acquisition device includes: acquisition module 310, calculation module 320 and data acquisition module 330.

[0101] The acquisition module 310 is configured to acquire the semantic representation vector of each statement in a plurality of statements based on the target semantic representation model. The calculation module 320 is configured to acquire the similarity value between any statement and the remaining statements in the plurality of statements based on the semantic representation vector and a similarity algorithm, resulting in multiple similarity values, wherein the remaining statements constitute a reference statement for any given statement. The data acquisition module 330 is configured to, if the relationship between the multiple similarity values ​​confirms that any given statement is similar to any reference statement, and confirms that the given statement and the reference statement do not belong to the same statement cluster, then confirm that the given statement and the reference statement constitute a set of negative sample data.

[0102] In some embodiments of this application, the calculation module 320 is configured to: form a statement pair with any one of the plurality of statements and the remaining statements respectively, thereby obtaining a plurality of statement pairs, wherein any statement pair includes the stated statement and a reference statement of the stated statement; read two semantic representation vectors corresponding to any statement pair from the storage unit, wherein the storage unit is used to store the semantic representation vectors of each statement in the plurality of statements; and obtain a similarity value for any statement pair based on the two semantic representation vectors and the similarity algorithm.

[0103] In some embodiments of this application, the data acquisition module 330 is configured to: sort the plurality of similarity values ​​according to a preset rule to obtain a sorting queue; and confirm that any statement is similar to any reference statement based on the position of the similarity value corresponding to any reference statement in the sorting queue.

[0104] In some embodiments of this application, the similarity algorithm adopts the cosine similarity algorithm. The data acquisition module 330 can also be configured to: sort the similarity values ​​in descending order; the step of confirming that any statement is similar to any reference statement based on the position of the similarity value corresponding to any reference statement in the sorting queue includes: if it is confirmed that the similarity value corresponding to any reference statement is located in the sorting queue before the set position, then the statement is confirmed to be similar to the reference statement.

[0105] In some embodiments of this application, the similarity algorithm adopts the cosine similarity algorithm, and the data acquisition module 330 can also be configured to: sort the similarity values ​​in ascending order; the step of confirming that any statement is similar to any reference statement based on the position of the similarity value corresponding to any reference statement in the sorting queue includes: if it is confirmed that the similarity value corresponding to any reference statement is located after the set position in the sorting queue, then it is confirmed that any statement is similar to the reference statement.

[0106] In some embodiments of this application, Figure 5 The training data acquisition device may further include a training module (not shown in the figure), which is configured to preprocess the acquired raw sentences to obtain preprocessed data. The preprocessing includes removing noise from the raw sentences and / or segmenting sentences with a length greater than a set threshold from the raw sentences; dividing the preprocessed data into a training dataset and a validation dataset; training the constructed initial semantic representation model based on the data in the training dataset to obtain a semantic representation model to be validated; and confirming that the semantic representation model to be validated passes validation based on the validation dataset to obtain the target semantic representation model.

[0107] In some embodiments of this application, the data acquisition module 330 may also be configured to: obtain all similar statements of each standard statement according to a clustering algorithm to obtain a statement cluster corresponding to each standard statement; or, cluster all statements into multiple statement clusters according to a clustering algorithm.

[0108] In some embodiments of this application, the similarity algorithm includes any one of the cosine similarity algorithm, Jaccard similarity algorithm, and Pearson correlation algorithm.

[0109] Some embodiments of this application also provide a training apparatus for a text matching model, the apparatus comprising one or more modules configured to employ... Figure 2 The negative sample data obtained by the method in any of the embodiments is used to train the text matching model.

[0110] Some embodiments of this application also provide a system comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform... Figure 2 The operation of the method in any of the embodiments.

[0111] Some embodiments of this application also provide one or more computer storage media for storing instructions, which, when executed by one or more computers, cause the one or more computers to perform... Figure 2 The operation of the method in any of the embodiments.

[0112] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0113] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0114] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for acquiring training data, characterized in that, The acquisition method includes: Obtain the semantic representation vector of each statement in multiple statements based on the target semantic representation model; Based on the semantic representation vector and similarity algorithm, the similarity value between any statement among the plurality of statements and the remaining statements is obtained, resulting in multiple similarity values, wherein the remaining statements constitute the reference statement of any statement. If, based on the magnitude of the multiple similarity values, it is confirmed that any statement is similar to any reference statement, and it is confirmed that any statement and any reference statement do not belong to the same statement cluster, then it is confirmed that any statement and the reference statement form a set of negative sample data; the statement cluster is obtained by using a clustering algorithm to acquire all similar statements of each standard statement, thus obtaining the statement cluster corresponding to each standard statement; or, all statements are clustered into multiple statement clusters using a clustering algorithm; each standard question represents the specific semantics of the corresponding statement cluster; The step of obtaining the similarity value between any statement and the remaining statements in the plurality of statements based on the semantic representation vector and the similarity algorithm includes: forming a statement pair by combining any statement in the plurality of statements with the remaining statements to obtain a plurality of statement pairs, wherein any statement pair in the plurality of statement pairs includes the statement and a reference statement of the statement; reading two semantic representation vectors corresponding to any statement pair in the plurality of statement pairs from a storage unit, wherein the storage unit is used to store the semantic representation vectors of each statement in the plurality of statements; and obtaining the similarity value of any statement pair based on the two semantic representation vectors and the similarity algorithm.

2. The acquisition method according to claim 1, characterized in that, The step of determining whether any statement is similar to any reference statement based on the magnitude relationship of the multiple similarity values ​​includes: The multiple similarity values ​​are sorted according to a preset rule to obtain a sorting queue; Based on the position of the similarity value corresponding to any of the reference statements in the sorting queue, it is determined that any statement is similar to any of the reference statements.

3. The acquisition method according to claim 2, characterized in that, The step of sorting the multiple similarity values ​​according to a preset rule includes: Sort the similarity values ​​in descending order; The step of determining whether a statement is similar to any reference statement based on its position in the sorting queue according to the similarity value corresponding to any reference statement includes: If it is confirmed that the similarity value corresponding to any of the reference statements is located in the sorting queue before the set position, then it is confirmed that any of the statements is similar to the reference statements.

4. The acquisition method according to claim 2, characterized in that, The step of sorting the multiple similarity values ​​according to a preset rule includes: Sort the similarity values ​​in ascending order; The step of determining whether a statement is similar to any reference statement based on its position in the sorting queue according to the similarity value corresponding to any reference statement includes: If it is confirmed that the similarity value corresponding to any of the reference statements is located after the set position in the sorting queue, then it is confirmed that any of the statements is similar to the reference statements.

5. The acquisition method according to claim 1, characterized in that, Before obtaining the semantic representation vectors of each statement in the multiple statements based on the target semantic representation model, the acquisition method further includes: The acquired raw statements are preprocessed to obtain preprocessed data, wherein the preprocessing includes removing noise from the raw statements and / or segmenting statements in the raw statements whose length is greater than a set threshold. The preprocessed data is divided into a training dataset and a validation dataset; The initial semantic representation model is trained based on the data in the training dataset to obtain the semantic representation model to be verified. If the semantic representation model to be verified is confirmed to pass the verification based on the verification dataset, then the target semantic representation model is obtained.

6. A device for acquiring training data, characterized in that, The apparatus is used to perform the method as described in claim 1, comprising: The acquisition module is configured to acquire the semantic representation vector of each statement in multiple statements based on the target semantic representation model; The calculation module is configured to obtain the similarity value between any one of the multiple statements and the remaining statements based on the semantic representation vector and the similarity algorithm, thereby obtaining multiple similarity values, wherein the remaining statements constitute the reference statement of any one of the statements; The data acquisition module is configured to confirm that any statement and the reference statement form a set of negative sample data if the relationship between the multiple similarity values ​​is confirmed to be similar to any reference statement, and it is confirmed that the any statement and the reference statement do not belong to the same statement cluster.

7. A computer system comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers perform operations according to any one of claims 1-5.

8. A computer storage medium storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1-5.

Images