Training methods, training systems, and text label determination methods for text matching models

Abstract

This disclosure presents a training method, training system, and text label determination method for a text matching model. One specific implementation of the training method includes: inputting labeled data and unlabeled text data from the training data into a pre-trained interactive model to obtain first predicted label data for the unlabeled text data; generating first new label data based on the first predicted label data and the labeled data; and fine-tuning the pre-trained text matching model according to the text data from the training data and the first new label data, wherein the text matching model adopts a dual-tower model structure. This implementation relates to model training techniques, utilizing the deep semantic expression capabilities of the interactive model to guide the text matching model in establishing a more accurate text labeling model. While ensuring the model's data processing efficiency, it can also improve the model's processing performance.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of model training technology, specifically to training methods, training systems, and text label determination methods for text matching models. Background Technology

[0002] Text and label matching falls under the category of sentence semantic similarity learning in Natural Language Processing (NLP). The essence of text matching is calculating the semantic similarity of texts, i.e., semantic matching. Currently, there are two main solutions: the Bi-Encoder approach and the Cross-Encoder approach. In a Bi-Encoder, two encoders are typically used to encode the query and document into vectors, and finally, a relevance discriminant function is used to calculate the similarity between the two vectors. In contrast, a Cross-Encoder usually concatenates the two sentences and inputs them into the encoder at once, outputting the semantic scores of the two sentences.

[0003] However, the inventors discovered that while the Bi-Encoder is more efficient at information retrieval in text matching, and its output can be directly used downstream, it cannot fully utilize the knowledge of the pre-trained model (BERT) due to the lack of information interaction. This typically results in a significant performance gap between it and the Cross-Encoder.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide training methods, training devices, model training systems, text label determination methods, electronic devices, computer-readable media, and computer program products for text matching models to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a method for training a text matching model, comprising: inputting labeled data and unlabeled text data from training data into a pre-trained interactive model to obtain first predicted label data for the unlabeled text data, wherein the interactive model is used to determine the semantic similarity between each input text data and each label data; generating first new label data based on the first predicted label data and the label data; and fine-tuning the pre-trained text matching model according to the text data and the first new label data from the training data, wherein the text matching model adopts a dual-tower model structure.

[0008] In some embodiments, generating first new label data based on first predicted label data and label data includes: dividing the first predicted label data into positive and negative samples according to the comparison result of the semantic similarity between the first predicted label data and unlabeled text data and the corresponding preset similarity threshold; and using the first predicted label data and label data as first new label data according to the division result.

[0009] In some embodiments, the method further includes: inputting label data and test text data into a pre-trained text matching model to obtain the semantic similarity between the test text data and each label data; determining the similarity distribution between each label data and each category according to the category to which the test text data belongs; and determining a preset similarity threshold between each label data and each category based on the similarity distribution and using a search strategy.

[0010] In some embodiments, the pre-trained interaction model is trained by: inputting labeled data and unlabeled text data into the pre-trained text matching model to obtain second predicted label data for the unlabeled text data; generating second new label data based on the second predicted label data and the labeled data; and initializing the interaction model based on the text data and the second new label data in the training data.

[0011] In some embodiments, the method further includes: in response to the completion of text matching model fine-tuning, regenerating second new label data based on the adjusted text matching model and training data to fine-tune the interaction model; and in response to the completion of interaction model fine-tuning, regenerating first new label data based on the adjusted interaction model and training data to continue fine-tuning the adjusted text matching model until the knowledge distillation iteration of the model is completed.

[0012] In some embodiments, during the knowledge distillation iteration process, the text matching model uses the mean squared error loss function, and the interaction model uses the cross-entropy loss function.

[0013] In some embodiments, fine-tuning a pre-trained text matching model includes: fine-tuning the pre-trained text matching model using a cue learning method.

[0014] In some embodiments, the pre-trained text matching model is trained by the following method: initializing the text matching model using text data and label data in the training data, and using the cross-entropy loss function, wherein the text matching model adopts a dual-tower model structure based on cue learning.

[0015] Secondly, some embodiments of this disclosure provide a training apparatus for a text matching model, comprising: a first predicted label determination unit configured to input label data and unlabeled text data from training data into a pre-trained interactive model to obtain first predicted label data for the unlabeled text data, wherein the interactive model is used to determine the semantic similarity between each input text data and each label data; a first new label generation unit configured to generate first new label data based on the first predicted label data and the label data; and a text matching model fine-tuning unit configured to fine-tune the pre-trained text matching model according to the text data and the first new label data from the training data, wherein the text matching model adopts a dual-tower model structure.

[0016] In some embodiments, the first new label generation unit is further configured to divide the first predicted label data into positive and negative samples based on the comparison result of the semantic similarity between the first predicted label data and the unlabeled text data and the corresponding preset similarity threshold; and to use the first predicted label data and the label data as the first new label data based on the division result.

[0017] In some embodiments, the training apparatus further includes a threshold determination unit, configured to input label data and test text data into a pre-trained text matching model to obtain the semantic similarity between the test text data and each label data; determine the similarity distribution between each label data and each category according to the category to which the test text data belongs; and determine a preset similarity threshold between each label data and each category using a search strategy based on the similarity distribution.

[0018] In some embodiments, the training apparatus further includes an interaction model initial training unit configured to input labeled data and unlabeled text data into a pre-trained text matching model to obtain second predicted label data for the unlabeled text data; generate second new label data based on the second predicted label data and the labeled data; and initialize the interaction model training based on the text data and the second new label data in the training data.

[0019] In some embodiments, the training apparatus further includes: an interaction model fine-tuning unit configured to, in response to the completion of text matching model fine-tuning, regenerate second new label data based on the adjusted text matching model and training data to fine-tune the interaction model; and a text matching model fine-tuning unit further configured to, in response to the completion of interaction model fine-tuning, regenerate first new label data based on the adjusted interaction model and training data to continue fine-tuning the adjusted text matching model until the knowledge distillation iteration of the model is completed.

[0020] In some embodiments, during the knowledge distillation iteration process, the text matching model uses the mean squared error loss function, and the interaction model uses the cross-entropy loss function.

[0021] In some embodiments, the text matching model fine-tuning unit is further configured to employ a cue learning method to fine-tune a pre-trained text matching model.

[0022] In some embodiments, the training apparatus further includes a text matching model initial training unit configured to initialize the text matching model using text data and label data in the training data, and to employ a cross-entropy loss function, wherein the text matching model employs a dual-tower model structure based on cue learning.

[0023] Thirdly, some embodiments of this disclosure provide a model training system, including: a first server on which a dual-tower model is installed, the dual-tower model being obtained using the training method described in any of the implementations of the first aspect; and a second server on which an interactive model is installed for iterative training with the dual-tower model using knowledge distillation.

[0024] Fourthly, some embodiments of this disclosure provide a text label determination method, including: in response to receiving text data to be analyzed, determining the text data to be analyzed as target text data; inputting the target text data and preset label data into a text matching model, and outputting predicted label data of the target text data, wherein the text matching model is obtained by the training method described in any of the implementations of the first aspect above; and determining and outputting label data of the target text data based on the predicted label data.

[0025] Fifthly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the methods described in any of the implementations of the first or fourth aspects above.

[0026] Sixthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the implementations of the first or fourth aspect above.

[0027] In a seventh aspect, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first or fourth aspect above.

[0028] The various embodiments of this disclosure have the following beneficial effects: The training methods in some embodiments of this disclosure obtain first predicted label data for some unlabeled text data through an interactive model. This enriches the sample data, thereby helping to improve the prediction accuracy of the text matching model. Furthermore, since the semantic modeling capability of interactive models is generally much higher than that of dual-tower models, more diverse pseudo-label data (i.e., predicted label data) can be obtained by utilizing the powerful semantic modeling capability of interactive models. Since the predicted label data is obtained through model prediction and is not the original label data in the sample data, it can be called pseudo-label data. This allows the text matching model to acquire and learn more effective information from the semantic retrieval (i.e., label attachment) task between text and labels, thereby further improving the model's prediction accuracy. In other words, the training method of this disclosure enables the final text matching model to possess both the deep semantic expression capability of interactive models and retain efficient information retrieval capabilities, thus achieving a win-win effect. Attached Figure Description

[0029] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0030] Figure 1 This is a flowchart of some embodiments of the training method disclosed herein;

[0031] Figure 2A This is a schematic diagram of the structure of some embodiments of the interaction model disclosed herein;

[0032] Figure 2B These are schematic diagrams of some embodiments of the dual-tower model disclosed herein;

[0033] Figure 3 These are flowcharts of some other embodiments of the training method disclosed herein;

[0034] Figure 4A These are schematic diagrams illustrating some application scenarios of the training method disclosed herein;

[0035] Figure 4B These are schematic diagrams illustrating other application scenarios of the training method disclosed herein;

[0036] Figure 5 These are schematic diagrams illustrating the structure of some embodiments of the training device disclosed herein;

[0037] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0038] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0039] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0040] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0041] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0042] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0043] Figure 1 A flow 100 illustrating some embodiments of a training method for a text matching model according to the present disclosure is shown. The training method includes the following steps:

[0044] Step 101: Input the labeled data and unlabeled text data from the training data into the pre-trained interactive model to obtain the first predicted label data for the unlabeled text data.

[0045] In some embodiments, the entity executing the training method of the text matching model (e.g., a model training server) can be connected to other electronic devices (such as wired or wireless connections) via wired or wireless connections. Figure 4BThe first server 401 and the second server 402 shown in the diagram communicate with each other. Here, the executing entity can input the labeled data and unlabeled text data from the training data into the pre-trained interactive model. Then, semantic similarity analysis is performed through the interactive model to obtain the first predicted label data for the unlabeled text data. The interactive model can be used to determine the semantic similarity between each input text data and each label data. This interactive model is the Cross-Encoder model mentioned above. 。

[0046] Understandably, the interaction model can determine the semantic similarity between each input text data and each tag data. Furthermore, it can determine the first predicted tags for that text data based on a specified number of tags with the highest similarity, in descending order of similarity. This specified number can be set according to the actual situation, such as 5 tags.

[0047] In some embodiments, the structure of the interaction model can be as follows: Figure 2A As shown. From Figure 2A As can be seen, the input layer of the interaction model can concatenate text data and label data (i.e., In...). x 1 to In y N y The input is fed into the encoder (i.e., the Encoder) in one go. The output of the encoder yields the representation vector (i.e., the Out). x 1 to Out y N y Next, through information aggregation (i.e., the aggregator), candidate vectors (i.e., candidate embeddings) can be obtained. Finally, through dimensionality reduction (i.e., dimensionality reduction), the semantic similarity score (i.e., score) is output.

[0048] Step 102: Generate first new label data based on the first predicted label data and the label data.

[0049] In some embodiments, based on the first predicted label data obtained in step 101 and the previously existing label data, the executing entity can generate first new label data. For example, the executing entity can use the aforementioned unlabeled text data and the corresponding first predicted label data as positive sample data to update the original training data. That is, the first predicted label data and the existing label data are used as new label data to obtain the first new label data.

[0050] In some embodiments, since the model's prediction results may be biased, the executing entity can correct the first predicted label data. For example, regarding the semantic similarity between the first predicted label data and unlabeled text data, the executing entity can classify the first predicted label data into positive and negative samples based on a comparison with a corresponding preset similarity threshold. For instance, if the semantic similarity of the first predicted label data is greater than the preset similarity threshold, then the first predicted label data can be used as a positive sample label for the corresponding text data. Conversely, if the semantic similarity of the first predicted label data is less than the preset similarity threshold, then the first predicted label data can be used as a negative sample label for the corresponding text data.

[0051] Next, based on the partitioning results, the executing entity can use the first predicted label data and the original label data as the first new label data. In other words, based on the positive and negative sample partitioning results, the executing entity can update the original positive label dataset with the first predicted label data of the positive samples, and update the original negative label dataset with the first predicted label data of the negative samples. Then, the updated label dataset and the label data in both the positive and negative label datasets are used as the first new label data.

[0052] It should be noted that the aforementioned preset similarity threshold can be pre-set. As an example, in real-world business scenarios, each text typically associates with at least one third-level category. In this case, the correction of predicted tag data can be achieved by setting a similarity threshold between each tag data and each third-level category.

[0053] Optionally, the preset similarity threshold can also be set by... Figure 3 The method shown is used to obtain the results; for details, please refer to [link / reference]. Figure 3 The relevant descriptions in the embodiments will not be repeated here.

[0054] Step 103: Fine-tune the pre-trained text matching model based on the text data and the first new label data in the training data.

[0055] In some embodiments, the executing entity can fine-tune a pre-trained text matching model based on the text data in the training data and the first new label data obtained in step 102. As an example, the executing entity can input the text data and the first new label data into the text matching model. Then, based on the label data predicted by the model and the first new label data corresponding to the text data, a loss function analysis can be performed to fine-tune the text matching model. Here, fine-tuning typically involves using a pre-trained model with one's own data, utilizing the weight parameter values ​​of the pre-trained model, and modifying the parameters of the last layer and the output category to train a model suitable for one's own data.

[0056] Understandably, during fine-tuning, the text matching model can employ a mean squared error loss function. This loss function helps the text matching model accurately fit the pseudo-scores of the interaction model, enabling it to learn deeper semantic knowledge.

[0057] Here, the text matching model can adopt a dual-tower model structure. As an example, such as... Figure 2B As shown, a text matching model can include a context encoder and a candidate label encoder. In this case, the input layer can process text data (i.e., In...) x 1 to In x N x The input text encoder produces a text representation vector (i.e., the Output vector). x 1 to Out x N x Afterwards, text information can be aggregated (i.e., a context aggregator) to obtain text vectors (i.e., context embeddings). Additionally, the input layer can process label data (i.e., In...). y 1 to In y N y The input to the candidate label encoder yields the label representation vector (i.e., the Out vector). y 1 to Out y N y Next, label information can be aggregated (i.e., CandidateAggregator) to obtain label vectors (i.e., Candidate Embedding). Finally, the similarity score between the output text vector and the label vector can be calculated (i.e., Score).

[0058] In some application scenarios, the executing agent can employ prompt learning to fine-tune a pre-trained text matching model. It's important to note that prompt learning is generally an effective method for adapting a pre-trained model to a specific task. By adding "hints" to the input without significantly altering the structure and parameters of the pre-trained language model, the true capabilities of the pre-trained model can be better utilized, thereby improving the model's zero-shot and few-shot learning abilities.

[0059] Prompt learning typically processes input text information according to a specific template, refactoring the task into a form that better utilizes the pre-trained language model. For example, a traditional supervised learning task trains a model P(y|x) to predict y from x as input. Prompt learning, however, relies on a pre-trained language model. By introducing a suitable template, the input x is reshaped into a fill-in-the-blank format x'. The reshaped input x' typically contains some empty slots. The language model then fills these slots to infer the corresponding y.

[0060] In some embodiments, the executing entity may employ a prompt learning method to add K randomly initialized tokens, i.e., prompt tokens, to the tag tokens. Their vector parameters can be trained. As an example, suppose that the text tokens obtained by the tokenizer are S[1], S[2], ..., S[M]. The tags obtained by the tokenizer are T = (T[1], T[2], ..., T[N]). The tag tokens combined with the prompt tokens can be T = (P[1], P[2], ..., P[K], T[1], T[2], ..., T[N]).

[0061] It is understood that, in this embodiment of the disclosure, a fine-tuning method based on a pre-trained language model (BERT, Bidirectional Encoder Representation from Transformers) can be used to fine-tune the text matching model. Here, the pre-trained language model typically no longer uses traditional unidirectional language models or shallow concatenation of two unidirectional language models for pre-training. Instead, a new masked language model (MLM) is employed to generate deep bidirectional language representations. Therefore, adding prompt tokens helps to further mine and utilize the knowledge of the pre-trained language model, improving the semantic retrieval performance of the text matching model (dual-tower model) on text-tags.

[0062] It's important to note that in practical applications, the most accurate semantic similarity method is often the interactive model. However, the retrieval efficiency of the interactive model is far lower than that of the dual-tower model. Therefore, in industry, the dual-tower model is generally preferred for solving sentence semantic similarity problems. However, the main drawback of the dual-tower model is its excessive separation between the query and document sides, with interaction only occurring in the final relevance discriminant function. That is, before inputting into the model, the query and document are completely independent, with no information exchange. This results in the model's representation generally being a static vector, leading to limitations in the learned semantic expression. In contrast, the interactive model fully perceives and interacts with the query and document at the feature level, enabling the extraction of richer semantic information. Therefore, the interactive model performs significantly better than the dual-tower model.

[0063] Furthermore, supervised learning training of text matching models typically requires a large amount of sample text data and corresponding sample label data, i.e., "text-label" pairs. The model's parameters are adjusted by analyzing the sample label data and the model's predicted label data to calculate the loss function. However, in some practical applications, the amount of sample data is often limited, and obtaining it is costly.

[0064] Based on the above description, the training methods in some embodiments of this disclosure obtain first predicted label data for some unlabeled text data through an interactive model. This enriches the sample data, thereby helping to improve the prediction accuracy of the text matching model. Furthermore, since the semantic modeling capability of interactive models is generally much higher than that of dual-tower models, more diverse pseudo-label data (i.e., predicted label data) can be obtained by utilizing the powerful semantic modeling capability of interactive models. Since the predicted label data is obtained through model prediction and is not the original label data in the sample data, it can be called pseudo-label data. This allows the text matching model to acquire and learn more effective information from the semantic retrieval (i.e., label attachment) task between text and labels, thereby further improving the model's prediction accuracy. In other words, the training method of this disclosure enables the final text matching model to possess both the deep semantic expression capability of interactive models and retain efficient information retrieval capabilities, thus achieving a win-win effect.

[0065] It should be noted that the pre-trained text matching model and pre-trained interaction model mentioned above can be obtained from public resources such as the internet, and are already initialized and trained models. This can improve the overall training efficiency.

[0066] Optionally, in order to meet the needs of real-world scenarios and improve the accuracy of model predictions, the pre-trained text matching model and the pre-trained interaction model mentioned above can also be obtained by initializing and training with the training data disclosed herein.

[0067] In some embodiments, the pre-trained text matching model can be trained using the following method: initializing the text matching model using text and label data from the training data, and employing a cross-entropy loss function. The text matching model here can be a dual-tower model. Furthermore, to improve the initial training effect of the text matching model, a cue-based learning method can also be used for model training and parameter tuning.

[0068] The pre-trained interaction model can be trained using the following method: input labeled data and unlabeled text data into the previously pre-trained text matching model to obtain the second predicted label data for the unlabeled text data; generate the second new label data based on the second predicted label data and the labeled data; and initialize the interaction model based on the text data and the second new label data in the training data.

[0069] Here, the process of initializing and training the interactive model can also be found in [reference needed]. Figure 1 The relevant descriptions of steps 101 to 103 in the embodiments are as follows. Since the semantic modeling capability of the interactive model is far superior to that of the dual-tower model, directly training the interactive model on pseudo-label data may lead to the interactive model severely fitting the erroneous information of the pseudo-label data, causing errors in the method of information processing during iteration, making the distillation iteration unable to achieve the expected results. Therefore, a preset similarity threshold can also be used here to correct the second predicted label data. Furthermore, during the training of the interactive model, cue learning can also be used to adjust the model parameters. The interactive model here can use a cross-entropy loss function (such as a binary classification cross-entropy loss function). This can prevent the interactive model from overfitting the pseudo-scores of the text matching model.

[0070] Understandably, a pre-trained text matching model (dual-tower model) is used to generate pseudo-label data to train an interaction model. Since the amount of generated label data is relatively small, training the interaction model will not take too long. Simultaneously, a desired interaction model can be obtained. Compared to using a general interaction model, the interaction model obtained by the method disclosed in this paper can improve the model's prediction accuracy in this domain. It also helps to improve the prediction performance of the subsequently fine-tuned text matching model.

[0071] Continue to refer to Figure 3 The diagram illustrates a flow 300 of yet another embodiment of the training method according to the present disclosure. The training method includes the following steps:

[0072] Step 301: Input the label data and test text data into the pre-trained text matching model to obtain the semantic similarity between the test text data and each label data.

[0073] In some embodiments, such as Figure 4A As shown, the execution entity of the text matching model training method can divide the <text-label> pairs into training and testing datasets according to a certain ratio. After initially training the text matching model using the <text-label> training dataset, the execution entity can perform inference on the <text-label> testing and validation set to obtain the similarity distribution between text and labels. Specifically, the execution entity can input the label data and test text data into the pre-trained text matching model to obtain the semantic similarity between the test text data and each label data.

[0074] Step 302: Determine the similarity distribution between each tag data and each category based on the category to which the test text data belongs.

[0075] In some embodiments, the executing entity can determine the similarity distribution between each tag data and each category based on the category to which the test text data belongs. It can be understood that the similarity between text and tags is the similarity between the category to which the text belongs and the tag. The category can be set according to the actual situation. As an example, each test text data can be associated with at least one third-level category, such as mobile phone, shampoo, etc. Based on the semantic similarity between the predicted output test text and tags, and the third-level categories associated with the test text, the similarity distribution of each tag on the third-level categories can be obtained.

[0076] Step 303: Based on the similarity distribution, use a search strategy to determine the preset similarity threshold between each tag data and each category.

[0077] In some embodiments, the executing entity can utilize a search strategy to determine a preset similarity threshold between each tag data and each category based on the similarity distribution. As an example, firstly, based on the true values ​​of the test text (i.e., the truly labeled tag data), the executing entity can obtain the true relationship distribution between each tag data and each category. Then, for each tag data, the executing entity can select an appropriate similarity threshold for each category based on the similarity distribution. This ensures that the overall accuracy of tag affixing reaches the target value (e.g., 0.9).

[0078] For example, the tag "drum set recommendation" may have a semantic match with text in three third-level categories: [percussion toys, drum set / jazz drum set, early childhood education]. Because the text themes differ across categories, the optimal similarity threshold for the tag is often inconsistent across different third-level categories to ensure a semantic matching accuracy higher than 0.9 for each category. For instance, the semantic similarity thresholds for percussion toys, drum set / jazz drum set, and early childhood education might be 0.6, 0.5, and 0.8, respectively.

[0079] The search strategy here is not limited and can include greedy algorithms and / or pruning, etc. As an example, the optimal similarity threshold for each tag across each category can be calculated using a greedy + search strategy. That is, as... Figure 4A As shown, by utilizing the third-level category (cid3) information of the text and employing a text cid3-label similarity threshold search strategy, a text cid3-label similarity threshold rule can be obtained. This threshold rule can be used to filter out erroneous information during the knowledge distillation iteration stage.

[0080] The training method in this embodiment further enriches and improves the process of generating preset similarity thresholds. Based on the initially trained text matching model, the semantic similarity between the output text and the label is predicted on the test set, combined with the true values ​​of the test set. Finally, threshold rules for the label and each category can be obtained. These threshold rules can be applied in the process of transferring data from the interaction model to the text matching model (or from the text matching model to the interaction model). This allows for data error correction of the output pseudo-label data, effectively reducing the transmission of erroneous information in the knowledge distillation iteration mechanism.

[0081] In some embodiments, the training method of this disclosure can utilize a text matching model (dual-tower model) and an interaction model to form a knowledge distillation iteration, thereby fine-tuning the text matching model. Specifically, in response to the completion of text matching model fine-tuning, a second set of new label data is regenerated based on the adjusted text matching model and training data to fine-tune the interaction model. And in response to the completion of interaction model fine-tuning, a first set of new label data is regenerated based on the adjusted interaction model and training data to continue fine-tuning the adjusted text matching model. This process is repeated until the model's knowledge distillation iteration is complete. In other words, the advantages of Cross-Encoder and Bi-Encoder can be leveraged to guide an accurate text-label semantic retrieval model in an unsupervised manner.

[0082] As an example, such as Figure 4B As shown, in the distillation iteration stage, pseudo-label data can first be generated using the Bi-Encoder attachment model (i.e., the text matching model). Then, using the threshold rules of labels and three-level categories, the pseudo-label data is corrected. Combined with existing supervised data, i.e., <text-label> pairs, the Cross-Encoder attachment model is trained and fine-tuned. For example, using the corrected pseudo-label data and existing label data, a binary classification cross-entropy loss function is used to fine-tune the Cross-Encoder that incorporates cue learning. The cross-entropy loss function can prevent the Cross-Encoder from overfitting the pseudo-scores of the Bi-Encoder.

[0083] At this point, the Cross-Encoder tagging model can leverage its powerful semantic modeling capabilities and more diverse pseudo-label data to extract more effective information from the tagging task. It should be noted that, because tagging models based on pre-trained language inherently possess a certain degree of fault tolerance, the impact of correctly tagged labels being incorrectly corrected to negative samples is negligible.

[0084] Conversely, pseudo-label data is generated using the obtained Cross-Encoder attachment model. Similarly, using the threshold rules of labels and three-level categories, the pseudo-label data is corrected, and combined with existing supervised data, the Bi-Encoder attachment model is fine-tuned. For example, using the corrected pseudo-label data and existing label data, the mean squared error (MSE) loss function is applied to the pre-trained Bi-Encoder for fine-tuning using cue learning. The MSE loss function helps the Bi-Encoder accurately fit the pseudo-scores of the Cross-Encoder. At this point, under the guidance of the Cross-Encoder and with more diverse pseudo-label data, the Bi-Encoder can further improve the label attachment effect.

[0085] Unlike traditional distillation tasks, this disclosed embodiment is based on supervised learning and employs the Bi-Encoder method. Addressing the lack of sentence interaction, it uses an iterative distillation approach with the Cross-Encoder. Specifically, it leverages data (existing labeled data and pseudo-labeled data) to transfer knowledge between the Bi-Encoder and Cross-Encoder. Utilizing the deep semantic representation capabilities of the Cross-Encoder, it guides the Bi-Encoder to build a more accurate text labeling model. Simultaneously, a label-based similarity threshold strategy is added during the distillation process to prevent the transmission of erroneous information. This further improves the transfer and utilization of effective information during distillation. The resulting Bi-Encoder model possesses both the deep semantic representation capabilities of the Cross-Encoder and retains efficient information retrieval capabilities.

[0086] Specifically, an iteration process can include distillation from Bi-Encoder to Cross-Encoder and distillation from Cross-Encoder to Bi-Encoder. Through this distillation iteration mechanism, the process can be repeated once or multiple times to continuously optimize encoder performance. In a topic tagging application for long text, the total number of tags is 712, and the text-tag pair data is 30,000 records. Using the method disclosed herein, a single distillation iteration improves the AUC (Area Under Curve) of the Bi-Encoder tagging model by 0.02 (0.825 > 0.845). After two iterations, the AUC improves by 0.034 (0.825 > 0.859).

[0087] As can be seen from the above description, the training method in this embodiment can achieve the following beneficial effects:

[0088] First, in supervised text semantic similarity retrieval, a more efficient Bi-Encoder is established by combining a Cross-Encoder with pseudo-label data. Compared with the traditional Bi-Encoder method, this method retains the high retrieval efficiency of the Bi-Encoder while incorporating some of the deeper semantic knowledge of the Cross-Encoder, achieving a win-win situation.

[0089] Secondly, during the knowledge distillation iteration process, information error correction through similarity thresholds can effectively prevent the transmission of erroneous information during the iteration process.

[0090] Furthermore, in semantic retrieval of text and labels, both Bi-Encoder and Cross-Encoder utilize prompt learning. Borrowing from the design of prompt learning in classification, prompt tokens are added to the input text of the label encoder. This approach can extract knowledge from the pre-trained language model itself, achieving better results than ordinary fine-tuning.

[0091] Further reference Figure 5 As a response to the above Figures 1 to 3 To implement the training method shown, this disclosure provides some embodiments of a training apparatus. These training apparatus embodiments are similar to... Figures 1 to 3 The methods and embodiments shown correspond to those described. This training device can be specifically applied to various electronic devices.

[0092] like Figure 5As shown, the training apparatus 500 in some embodiments may include: a first predicted label determination unit 501, configured to input label data and unlabeled text data from the training data into a pre-trained interactive model to obtain first predicted label data for the unlabeled text data, wherein the interactive model is used to determine the semantic similarity between each input text data and each label data; a first new label generation unit 502, configured to generate first new label data based on the first predicted label data and the label data; and a text matching model fine-tuning unit 503, configured to fine-tune the pre-trained text matching model according to the text data and the first new label data from the training data, wherein the text matching model adopts a dual-tower model structure.

[0093] In some embodiments, the first new label generation unit 502 may be further configured to divide the first predicted label data into positive and negative samples based on the comparison result of the semantic similarity between the first predicted label data and the unlabeled text data and the corresponding preset similarity threshold; and to use the first predicted label data and the label data as the first new label data based on the division result.

[0094] In some embodiments, the training device 500 may further include a threshold determination unit (not shown in the figure), configured to input label data and test text data into a pre-trained text matching model to obtain the semantic similarity between the test text data and each label data; determine the similarity distribution between each label data and each category according to the category to which the test text data belongs; and determine a preset similarity threshold between each label data and each category using a search strategy based on the similarity distribution.

[0095] In some embodiments, the training apparatus 500 may further include an interaction model initial training unit (not shown in the figure), configured to input labeled data and unlabeled text data into a pre-trained text matching model to obtain second predicted label data for the unlabeled text data; generate second new label data based on the second predicted label data and the labeled data; and initialize the interaction model training based on the text data and the second new label data in the training data.

[0096] In some embodiments, the training apparatus 500 may further include: an interaction model fine-tuning unit (not shown in the figure), configured to regenerate second new label data based on the adjusted text matching model and training data in response to the completion of text matching model fine-tuning, so as to fine-tune the interaction model; and a text matching model fine-tuning unit, further configured to regenerate first new label data based on the adjusted interaction model and training data in response to the completion of interaction model fine-tuning, so as to continue fine-tuning the adjusted text matching model until the knowledge distillation iteration of the model is completed.

[0097] In some embodiments, during the knowledge distillation iteration process, the text matching model uses the mean squared error loss function, and the interaction model uses the cross-entropy loss function.

[0098] In some embodiments, the text matching model fine-tuning unit 503 may be further configured to employ a cue learning method to fine-tune a pre-trained text matching model.

[0099] In some embodiments, the training apparatus 500 may further include a text matching model initial training unit (not shown in the figure), configured to initialize the text matching model using text data and label data in the training data, and to employ a cross-entropy loss function, wherein the text matching model adopts a dual-tower model structure based on cue learning.

[0100] It is understandable that the units described in the training device 500 are related to the reference. Figures 1 to 3 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the training device 500 and the units contained therein, and will not be repeated here.

[0101] This disclosure also provides a model training system. The model training system may include a first server (such as...). Figure 4B The first server 401 shown) and the second server (as shown) Figure 4B The second server shown is 402. A dual-tower model can be installed on the first server. The dual-tower model described above... Figures 1 to 3 In the embodiments, the training method described in any implementation is used. An interactive model can be installed on the second server. The interactive model can be used for iterative training with the dual-tower model through knowledge distillation. This allows the dual-tower model to possess both high retrieval efficiency and the deep semantic knowledge of the interactive model, improving the model's prediction performance.

[0102] In addition, this disclosure also provides a text tag determination method. This text tag determination method may include: in response to receiving text data to be analyzed, determining the text data to be analyzed as target text data; inputting the target text data and preset tag data into a text matching model, and outputting predicted tag data for the target text data, wherein the text matching model employs the above-mentioned... Figures 1 to 3 In the embodiments, the training method described in any implementation is used to obtain: based on the predicted label data, the label data of the target text data is determined and output.

[0103] In other words, in practical applications, the text data to be analyzed can be input into the pre-trained text matching model. The text data to be analyzed can be either unlabeled or labeled, to determine the accuracy of the current label. Here, the predicted label data output by the model can be directly used as the label data for the target text data. Alternatively, at least one predicted label data can be selected from multiple predicted label data sets to serve as the label data for that text data. The data output method is also unrestricted; it can be displayed on a terminal or the correspondence between the target text data and the label data can be stored.

[0104] It should be noted that the operations, features, and beneficial effects described above for the training method of the text matching model are also applicable to the text label determination method disclosed herein, and will not be repeated here.

[0105] The following is for reference. Figure 6 It shows a schematic diagram of the structure of an electronic device 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0106] like Figure 6 As shown, the terminal device 600 may include a processing unit 601 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the terminal device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0107] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, speakers, vibrators, etc.; storage devices 608 including, for example, disks, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0108] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0109] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0110] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0111] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: input labeled data and unlabeled text data from the training data into a pre-trained interactive model to obtain first predicted label data for the unlabeled text data, wherein the interactive model is used to determine the semantic similarity between each input text data and each label data; generate first new label data based on the first predicted label data and the label data; and fine-tune a pre-trained text matching model according to the text data in the training data and the first new label data, wherein the text matching model adopts a dual-tower model structure.

[0112] Alternatively, in response to receiving text data to be analyzed, the text data to be analyzed is identified as target text data; the target text data and preset label data are input into a text matching model, and the predicted label data of the target text data is output, wherein the text matching model is obtained by the training method described in any of the above embodiments; and the label data of the target text data is determined and output based on the predicted label data.

[0113] Furthermore, computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first predicted label determination unit, a first new label generation unit, and a text matching model fine-tuning unit. The names of these units do not necessarily limit the specific unit; for example, the first predicted label determination unit may also be described as "a unit that obtains first predicted label data for unlabeled text data."

[0116] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0117] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the training methods or text label determination methods described above.

[0118] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for training a text matching model, comprising: The labeled data and unlabeled text data in the training data are input into a pre-trained interaction model to obtain the first predicted label data of the unlabeled text data, wherein the interaction model is used to determine the semantic similarity between each input text data and each of the labeled data. Based on the first predicted label data and the label data, generate the first new label data; Based on the text data in the training data and the first new label data, the pre-trained text matching model is fine-tuned, wherein the text matching model adopts a dual-tower model structure. In response to the completion of the text matching model fine-tuning, a second new label data is generated based on the adjusted text matching model and the training data to fine-tune the interaction model. In response to the completion of the fine-tuning of the interaction model, the first new label data is regenerated based on the adjusted interaction model and the training data to continue fine-tuning the adjusted text matching model. This process is repeated until the knowledge distillation iteration of the model is completed.

2. The training method according to claim 1, wherein, The step of generating first new label data based on the first predicted label data and the label data includes: Based on the semantic similarity between the first predicted label data and the unlabeled text data, and the comparison result with the corresponding preset similarity threshold, the first predicted label data is divided into positive and negative samples. Based on the division results, the first predicted label data and the label data are used as the first new label data.

3. The training method according to claim 2, wherein, The method further includes: The label data and test text data are input into the pre-trained text matching model to obtain the semantic similarity between the test text data and each of the label data. Based on the category to which the test text data belongs, determine the similarity distribution between each tag data and each category; Based on the similarity distribution, a preset similarity threshold between each of the tag data and each category is determined using a search strategy.

4. The training method according to claim 1, wherein, The pre-trained interaction model was trained using the following method: The labeled data and the unlabeled text data are input into the pre-trained text matching model to obtain the second predicted label data of the unlabeled text data; Based on the second predicted label data and the label data, generate second new label data; The interaction model is initialized and trained based on the text data in the training data and the second new label data.

5. The training method according to claim 4, wherein, During the knowledge distillation iteration process, the text matching model uses the mean squared error loss function, and the interaction model uses the cross-entropy loss function.

6. The training method according to claim 1, wherein, The fine-tuning of the pre-trained text matching model includes: A cue-based learning approach is used to fine-tune a pre-trained text matching model.

7. The training method according to any one of claims 1-6, wherein, The pre-trained text matching model was trained using the following method: The text matching model is initialized and trained using the text data and the label data in the training data, and a cross-entropy loss function is used. The text matching model adopts a dual-tower model structure based on cue learning.

8. A training device for a text matching model, comprising: The first predictive label determination unit is configured to input the labeled data and unlabeled text data from the training data into a pre-trained interaction model to obtain the first predicted label data of the unlabeled text data, wherein the interaction model is used to determine the semantic similarity between each input text data and each of the label data. The first new label generation unit is configured to generate first new label data based on the first predicted label data and the label data; The text matching model fine-tuning unit is configured to fine-tune the pre-trained text matching model based on the text data in the training data and the first new label data, wherein the text matching model adopts a dual-tower model structure. The generation unit is configured to generate second new label data based on the adjusted text matching model and the training data in response to the completion of the text matching model fine-tuning, so as to fine-tune the interaction model. The fine-tuning unit is configured to, in response to the completion of the fine-tuning of the interaction model, regenerate the first new label data based on the adjusted interaction model and the training data, so as to continue to fine-tune the adjusted text matching model, and so on, until the knowledge distillation iteration of the model is completed.

9. A model training system, comprising: A first server is equipped with a dual-tower model, which is obtained using the training method described in any one of claims 1-7; The second server is equipped with an interactive model for iterative training of knowledge distillation with the dual-tower model.

10. A method for determining text labels, comprising: In response to receiving text data to be analyzed, the text data to be analyzed is identified as target text data; The target text data and preset label data are input into a text matching model, and the predicted label data of the target text data is output. The text matching model is obtained by training method as described in any one of claims 1-7. Based on the predicted label data, the label data of the target text data is determined and output.

11. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7 or 10.

12. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7 or 10.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7 or 10.

Images