Text representation model training method and device, storage medium and terminal

Abstract

The embodiment of the invention discloses a text representation model training method and device, a storage medium and a terminal. Constructing an initial text representation model; obtaining a plurality of sample statement pairs, wherein the sample statement pairs are statement pairs consisting of sample query statements and sample comparison statements; training an initial text representation model based on the plurality of sample statement pairs; in the training process of the initial text representation model, the initial text representation model is controlled to execute at least two training tasks for the multiple sample statement pairs, and task loss values corresponding to the training tasks are calculated according to execution results of the training tasks, and adjusting parameters of the initial text representation model based on each task loss value until the initial text representation model converges to obtain a trained text representation model. A three-level positive sample, weak positive sample and negative sample comparative learning mechanism is adopted, the model is made to predict various output targets, loss values for the multiple output targets are integrated into a loss function, and the learning efficiency of the model is improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular, to a method, device, storage medium, and terminal for training a text representation model. Background Art

[0002] All along, text representation has been playing a fundamental role in natural language processing. Especially in the era of large models, as a key part in the retrieval-augmented generation process, knowledge retrieval requires better text representation within a specific domain, which determines the accuracy and completeness of the answers generated by large prediction models. Therefore, the quality of text representation will significantly affect various language understanding-related performances in the model. However, in practical applications, when studying each specific task, the definitions of similarity, partial acceptability, and unacceptability are likely to be different. Therefore, it is necessary to "adjust" these models specifically to better match the specific criteria, special expressions, and different styles of each specific task. Therefore, it is necessary to perform fine-tuning training on the text representation model for a specific task, so that the model can fully learn the definitions and criteria of similarity, medium, and dissimilarity under the current task. Summary of the Invention

[0003] The embodiments of this specification provide a method, device, storage medium, and terminal for training a text representation model, which can solve the technical problem that the effect of model fine-tuning training in the related art is not ideal.

[0004] In a first aspect, the embodiments of this specification provide a method for training a text representation model, the method includes:

[0005] Construct an initial text representation model based on a base model;

[0006] Obtain a plurality of sample sentence pairs related to a target scenario, where the sample sentence pairs are sentence pairs composed of a sample query sentence and a sample comparison sentence, and the sample comparison sentence is a positive sample and / or negative sample and / or weak positive sample corresponding to the sample query sentence;

[0007] Train the initial text representation model based on the plurality of sample sentence pairs;

[0008] During the training process of the initial text representation model, control the initial text representation model to execute at least two training tasks for the plurality of sample sentence pairs, calculate the task loss value corresponding to each training task according to the execution result of each training task, and adjust the parameters of the initial text representation model based on each task loss value until the initial text representation model converges, obtaining a trained text representation model.

[0009] In a possible implementation, when the above-mentioned multiple sample statement pairs include positive example statement pairs where the above-mentioned sample comparison statement is the above-mentioned positive sample and negative example statement pairs where the above-mentioned sample comparison statement is the above-mentioned negative sample, the above-mentioned training task at least includes a contrast task; the method further includes: when performing the above-mentioned contrast task, controlling the above-mentioned initial text representation model to distinguish the positive sample and negative sample corresponding to the sample query statement in each sample statement pair for the above-mentioned multiple sample statement pairs respectively.

[0010] In a possible implementation, when the above-mentioned multiple sample statement pairs include positive example statement pairs where the above-mentioned sample comparison statement is the above-mentioned positive sample, negative example statement pairs where the above-mentioned sample comparison statement is the above-mentioned negative sample, and weakly positive example statement pairs where the above-mentioned sample comparison statement is the above-mentioned weakly positive sample, the above-mentioned training task at least includes a ranking task; the method further includes: when performing the above-mentioned ranking task, controlling the above-mentioned initial text representation model to rank the positive sample, negative sample, and weakly positive sample corresponding to the sample query statement in each sample statement pair in descending order of similarity to the above-mentioned sample query statement.

[0011] In a possible implementation, when the above-mentioned multiple sample statement pairs include positive example statement pairs where the above-mentioned sample comparison statement is the above-mentioned positive sample, negative example statement pairs where the above-mentioned sample comparison statement is the above-mentioned negative sample, and weakly positive example statement pairs where the above-mentioned sample comparison statement is the above-mentioned weakly positive sample, the above-mentioned training task at least includes a classification task; the method further includes: when performing the above-mentioned classification task, controlling the above-mentioned initial text representation model to perform label classification on the positive sample, negative sample, and weakly positive sample corresponding to the sample query statement in each sample statement pair.

[0012] In a possible implementation, the method further includes: selecting a loss weighting coefficient corresponding to each training task based on an ablation experiment; adjusting the parameters of the above-mentioned initial text representation model based on the loss value of each task until the above-mentioned initial text representation model converges, including: calculating the total loss value of the above-mentioned initial text representation model based on the loss weighting coefficient corresponding to each training task and the task loss value, and adjusting the parameters of the above-mentioned initial text representation model based on the above-mentioned total loss value until the above-mentioned initial text representation model converges.

[0013] In a possible implementation manner, obtaining multiple sample statement pairs related to a target scenario includes: obtaining multiple initial sample query statements according to a preset data set corresponding to the target scenario, and multiple initial positive samples and / or multiple initial negative samples and / or multiple initial weakly positive samples corresponding to the multiple initial sample query statements; performing a positive sample amplification operation on an initial sample query statement without an initial positive sample; performing a negative sample amplification operation on an initial sample query statement without an initial negative sample; and performing a weakly positive sample amplification operation on an initial sample query statement without an initial weakly positive sample.

[0014] In a possible implementation manner, the positive sample amplification operation at least includes random repetition, the negative sample amplification operation at least includes random extraction, and the weakly positive sample amplification operation includes at least one of syntax tree rewriting, random replacement, random insertion, random shuffling, and positive and negative sample splicing.

[0015] In a possible implementation manner, the positive sample amplification operation and / or the negative sample amplification operation and / or the weakly positive sample amplification operation is implemented based on a preset unsupervised sample provider.

[0016] In a second aspect, an embodiment of this specification provides a training device for a text representation model. The device includes:

[0017] A model construction module, configured to construct an initial text representation model based on a base model;

[0018] A sample preparation module, configured to obtain multiple sample statement pairs related to a target scenario, where the sample statement pairs are statement pairs composed of a sample query statement and a sample comparison statement, and the sample comparison statement is a positive sample and / or a negative sample and / or a weakly positive sample corresponding to the sample query statement;

[0019] A model training module, configured to train the initial text representation model based on the multiple sample statement pairs;

[0020] A model learning module, configured to, during the training process of the initial text representation model, control the initial text representation model to execute at least two training tasks for the multiple sample statement pairs, calculate task loss values corresponding to the training tasks according to the execution results of the training tasks, and adjust parameters of the initial text representation model based on the task loss values until the initial text representation model converges, so as to obtain a trained text representation model.

[0021] In a third aspect, an embodiment of this specification provides a computer program product containing instructions. When the computer program product runs on a computer or a processor, the computer or the processor is enabled to execute the steps of the above method.

[0022] Fourthly, an embodiment of this specification provides a computer storage medium. The computer storage medium stores multiple instructions, and the instructions are adapted to be loaded and executed by a processor to perform the steps of the above method.

[0023] Fifthly, an embodiment of this specification provides a terminal, including a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is adapted to be loaded and executed by the processor to perform the steps of the above method.

[0024] The beneficial effects brought by the technical solutions provided by some embodiments of this specification at least include:

[0025] An embodiment of this specification provides a training method for a text representation model. An initial text representation model is constructed based on a basic model; multiple sample statement pairs related to a target scenario are obtained. The sample statement pairs are statement pairs composed of a sample query statement and a sample comparison statement, and the sample comparison statement is a positive sample and / or a negative sample and / or a weak positive sample corresponding to the sample query statement; the initial text representation model is trained based on the multiple sample statement pairs; during the training process of the initial text representation model, the initial text representation model is controlled to execute at least two training tasks for the multiple sample statement pairs, and task loss values corresponding to the training tasks are calculated according to the execution results of the training tasks. The parameters of the initial text representation model are adjusted based on the task loss values until the initial text representation model converges, and a trained text representation model is obtained. In the embodiment of this specification, a contrastive learning mechanism of three levels of positive samples, weak positive samples, and negative samples is adopted to construct a multi-level contrastive learning framework for the model. The processing ability of the model for complex tasks is optimized through the carefully divided sample categories. This hierarchical contrastive relationship prompts the model to deepen its understanding of the data in multiple dimensions. Further, this mechanism enables the model to predict multiple different types of output targets during the training process, enabling the model to accurately capture and learn the features of the samples in different aspects. The relevant features of a specific task are learned by optimizing the model through the hierarchical contrastive relationship, significantly improving the discrimination and robustness of the text representation. In the stage of adjusting the model parameters, the loss values obtained after predicting multiple output targets are integrated into a unified composite loss function, enabling the model to optimize the features in multiple directions under the target scenario in a single framework. This training and learning mechanism combining multiple output targets greatly improves the learning efficiency and generalization performance of the entire framework under a specific scenario / specific task. Description of the Drawings

[0026] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of this specification. For those skilled in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0027] Figure 1 Exemplary system architecture diagram of a training method for a text representation model provided by an embodiment of this specification;

[0028] Figure 2 Flow schematic diagram of a training method for a text representation model provided by an embodiment of this specification;

[0029] Figure 3 Flow schematic diagram of a training method for a text representation model provided by an embodiment of this specification;

[0030] Figure 4 Block diagram of a training device for a text representation model provided by an embodiment of this specification;

[0031] Figure 5 Structural schematic diagram of a terminal provided by an embodiment of this specification. Detailed implementation manners

[0032] To make the features and advantages of the embodiments of this specification more obvious and understandable, the following will clearly and completely describe the technical solutions in the embodiments of this specification in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only some embodiments of this specification, not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the embodiments of this specification.

[0033] When the following description involves the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of this specification. On the contrary, they are only examples of devices and methods consistent with some aspects of the embodiments of this specification detailed in the appended claims. And in the description of the embodiments of this specification, unless otherwise stated, " / " means "or". For example, A / B can represent A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this specification, "a plurality" means two or more than two.

[0034] Hereinafter, the terms "first" and "second" are for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0035] In the current era of large language models, text representation, as one of the core technologies in the field of natural language processing (NLP), is not only the foundation of language understanding but also a crucial component of advanced technologies such as Retrieval-Augmented Generation (RAG). In the RAG framework, the quality of the model's text representation determines whether the retrieval system can accurately find the most relevant information to the question from a vast amount of data, thus providing support for the large language model (LLM) to generate high-quality and accurate answers. The application scope of text representation is extremely wide, covering multiple tasks such as unsupervised text classification, text similarity assessment, clustering analysis, information retrieval, and re-ranking of retrieval results. For example, in the unsupervised classification task, the quality of text representation determines whether the model can automatically identify the potential categories of the text without labeled data; in text similarity assessment, text representation directly affects the model's judgment of the semantic similarity between two texts; in the information retrieval task, the quality of text representation determines whether the retrieval system can quickly find the most relevant documents to the query from a large-scale document library.

[0036] However, in practical applications, the definition of "similarity" may vary significantly under different specific scenarios / specific tasks, and some tasks may have specific definitions for "partial acceptability" or "unacceptability". To address these challenges, it is usually necessary to perform targeted fine-tuning training on the pre-trained text representation model so that the model can fully learn the similarity criteria, domain knowledge, etc. under specific tasks. For example, in the financial field, the text representation model may need to learn how to distinguish the subtle differences between different financial terms; in a multilingual environment, the model needs to adapt to the expression habits and cultural backgrounds between different languages. Through fine-tuning, the model can better capture the semantic features in specific tasks, thereby improving its performance in specific application scenarios.

[0037] Generally speaking, text representation mainly generates low-dimensional vector representations of texts of different lengths through the understanding of words. Common methods include TF-IDF, LSA, word2vec, and corresponding sentence vector calculation methods. Therefore, when training the text representation performance of a model, a common training method of contrastive learning is used to enable the model to generate text vectors that can accurately express semantics by aligning similar pairs and distinguishing irrelevant pairs. Therefore, in order to improve the effect of contrastive learning, common methods include enhancing positive sample pairs (such as CERT, ConSERT) or negative sample pairs (such as SimCSE, ESimCSE, DiffCSE, InfoCSE). However, these methods require retraining the text representation model from scratch, and their accuracy in calculating text similarity is acceptable, but their performance in tasks such as recall and ranking is not reliable enough. In addition, the basic model obtained through the above training method still needs to be fine-tuned specifically when used for specific tasks. The existing fine-tuning training methods are often cumbersome to execute, costly to train, and the performance gain cannot meet the performance requirements.

[0038] Therefore, the embodiments of this specification provide a training method for a text representation model to solve the technical problem that the effect of the fine-tuning training of the above model is not ideal.

[0039] Please refer to Figure 1 , Figure 1 which is an exemplary system architecture diagram of a training method for a text representation model provided by the embodiments of this specification.

[0040] As Figure 1 shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 is used to provide a medium for the communication link between the terminal 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links. For example, the wired communication links include optical fibers, twisted pairs, or coaxial cables, and the wireless communication links include Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0041] The terminal 101 can interact with the server 103 via the network 102 to receive messages from the server 103 or send messages to the server 103. Alternatively, the terminal 101 can interact with the server 103 via the network 102 to receive messages or data sent by other users to the server 103. The terminal 101 can be hardware or software. When the terminal 101 is hardware, it can be various electronic devices, including but not limited to tablet computers, laptop portable computers, and desktop computers, etc. When the terminal 101 is software, it can be installed in the above-listed electronic devices, and it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited herein.

[0042] In the embodiments of this specification, the terminal 101 first constructs an initial text representation model based on a basic model; in the training preparation stage, the terminal 101 obtains multiple sample statement pairs related to the target scenario. The sample statement pair is a statement pair composed of a sample query statement and a sample comparison statement, and the sample comparison statement is a positive sample and / or negative sample and / or weak positive sample corresponding to the sample query statement; after the samples and the model architecture are ready, the terminal 101 can train the initial text representation model based on the multiple sample statement pairs. Specifically, in the training process of the initial text representation model, control the initial text representation model to execute at least two training tasks for the multiple sample statement pairs, calculate the task loss value corresponding to each training task according to the execution results of each training task, and adjust the parameters of the initial text representation model based on each task loss value until the initial text representation model converges to obtain the trained text representation model.

[0043] The server 103 can be a business server that provides various services. It should be noted that the server 103 can be hardware or software. When the server 103 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or it can be implemented as a single server. When the server 103 is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited herein.

[0044] Alternatively, the system architecture may not include the server 103. In other words, the server 103 can be an optional device in the embodiments of this specification, that is, the method provided in the embodiments of this specification can be applied to a system structure that only includes the terminal 101, which is not limited in the embodiments of this specification.

[0045] It should be understood that Figure 1 the numbers of terminals, networks, and servers in

[0046] Please refer toFigure 2 , Figure 2 This is a schematic flowchart of a method for training a text representation model provided by an embodiment of this specification. The execution subject of the embodiment of this specification can be a terminal that executes the training of the text representation model, or a processor in the terminal that executes the training method of the text representation model, or a training service of the text representation model in the terminal that executes the training method of the text representation model. For the convenience of description, hereinafter, the execution subject is taken as an example of the processor in the terminal to introduce the specific execution process of the method for training the text representation model.

[0047] As Figure 2 shown, the method for training the text representation model can at least include:

[0048] S202. Construct an initial text representation model based on a base model.

[0049] Optionally, in different application scenarios, the text representation model needs to be fine-tuned for specific tasks in the scenario. Then, before performing fine-tuning training, a pre-trained base text representation model can be obtained first. The base model can output its text representation features based on the input data. Then, the initial text representation model constructed based on the base model also has the ability to output text representation features. After constructing the initial text representation model for a specific task scenario, the initial text representation model can be trained specifically.

[0050] S204. Obtain multiple sample sentence pairs related to the target scenario. The sample sentence pair is a sentence pair composed of a sample query sentence and a sample comparison sentence. The sample comparison sentence is the positive sample and / or negative sample and / or weak positive sample corresponding to the sample query sentence.

[0051] Optionally, the text representation model mainly relies on contrastive learning for training. Contrastive learning is an unsupervised learning technique. Its core idea is to learn the representation of data by comparing the similarities and differences between different samples. By designing a contrastive loss function (Contrastive Loss), the model can effectively capture the semantic structure of the data, so that in the representation space, similar samples are closer and dissimilar samples are farther apart. It does not rely on labeled data, but through the mutual relationship between samples, the model can learn meaningful feature representations. In contrastive learning, there is usually a positive sample pair and multiple negative sample pairs. A positive sample pair refers to a similar or related sample pair, while a negative sample pair is a dissimilar or unrelated sample pair. The goal of contrastive learning is to make the representations between positive sample pairs closer, while making the representations between negative sample pairs more distant.

[0052] In the embodiments of this specification, when preparing samples, multiple sample statement pairs related to the target scenario can be obtained. These sample statement pairs are composed of a sample query statement and a sample comparison statement. The sample query statement is the statement serving as the anchor sample, and the sample statement is the positive sample and / or negative sample and / or weakly positive sample corresponding to the sample query statement. Here, the embodiments of this specification consider that in addition to positive samples and negative samples, weakly positive samples can also be used to help the model learn to identify fine-grained semantic differences in the text. Specifically, the positive samples in the embodiments of this specification represent statements that are highly relevant or completely match the anchor query statement, and the two constitute the benchmark for the model to learn; the weakly positive samples are answers that, although not ideal, can provide some relevant information or partial help and are between positive samples and negative samples, and they provide a broader learning boundary for the model; while the negative samples are statements that are completely irrelevant or incorrect to the anchor query statement and are used to strengthen the discrimination ability of the model. Since these sample statement pairs are highly relevant to the target scenario, these samples can support the model to specifically and deeply learn various knowledge of specific prediction tasks in the target scenario, thereby realizing the fine-tuning of the model under the target scenario / target task.

[0053] S206. Train the initial text representation model based on multiple sample statement pairs.

[0054] S208. During the training process of the initial text representation model, control the initial text representation model to perform at least two training tasks on multiple sample statement pairs, calculate the task loss values corresponding to the respective training tasks according to the execution results of the respective training tasks, and adjust the parameters of the initial text representation model based on the respective task loss values until the initial text representation model converges to obtain the trained text representation model.

[0055] Optionally, after the sample sentence pairs are prepared, an initial text representation model can be trained based on multiple sample sentence pairs. During the training process of the initial text representation model, the embodiments of this specification use a multi-task joint optimization strategy for the model. Specifically, it is to control the initial text representation model to perform at least two training tasks on multiple sample sentence pairs. In different training tasks, the model will give prediction results of the samples on different output targets, and a specific loss function will be designed for each task. For example, if there is a contrastive learning task, a contrastive loss is adopted; if there is a classification task, a cross-entropy loss is adopted; if there is a ranking task, a contrastive loss is also adopted. During training, according to the execution results of the model in each training task and the loss function corresponding to the task, the task loss values corresponding to each training task are calculated respectively. Then, the loss values of these tasks are dynamically unified into a comprehensive loss function, that is, the parameters of the initial text representation model are adjusted based on the task loss values until the initial text representation model converges, and a trained text representation model is obtained. This cross-task deep feature learning mechanism enables the model to simultaneously optimize multiple task objectives in a single framework, not only improving the learning efficiency but also promoting knowledge transfer between different tasks.

[0056] It should be noted that the model end training conditions of the model can include, for example, that the value of the loss function is less than or equal to a preset loss function threshold, the number of iterations reaches a preset number threshold, etc. The specific model end training conditions can be determined based on the actual situation and are not specifically limited here.

[0057] In the embodiments of this specification, a training method for a text representation model is provided. An initial text representation model is constructed based on a basic model; multiple sample statement pairs related to a target scenario are obtained, where a sample statement pair is a statement pair composed of a sample query statement and a sample comparison statement, and the sample comparison statement is a positive sample and / or a negative sample and / or a weak positive sample corresponding to the sample query statement; the initial text representation model is trained based on the multiple sample statement pairs; during the training process of the initial text representation model, the initial text representation model is controlled to execute at least two training tasks for the multiple sample statement pairs, and according to the execution results of each training task, the task loss value corresponding to each training task is calculated, and the parameters of the initial text representation model are adjusted based on each task loss value until the initial text representation model converges, and a trained text representation model is obtained. In the embodiments of this specification, a contrastive learning mechanism of three-level positive samples, weak positive samples, and negative samples is adopted to construct a multi-level contrastive learning framework for the model, and the processing ability of the model for complex tasks is optimized through carefully divided sample categories. This hierarchical contrastive relationship prompts the model to deepen its understanding of the data in multiple dimensions. Further, this mechanism enables the model to predict multiple different types of output targets during the training process, enabling the model to accurately capture and learn the characteristics of the samples in different aspects, and optimizing the learning of relevant characteristics of the model for specific tasks through the hierarchical contrastive relationship, significantly improving the discrimination and robustness of the text representation. In the stage of model parameter tuning, the loss values obtained after predicting multiple output targets are integrated into a unified composite loss function, enabling the model to optimize the characteristics in multiple directions under the target scenario in a single framework. This training and learning mechanism combining multiple output targets greatly improves the learning efficiency and generalization performance of the entire framework in a specific scenario / specific task.

[0058] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of a training method for a text representation model provided by the embodiments of this specification.

[0059] As Figure 3 shown, the training method for the text representation model may at least include:

[0060] S302. Construct an initial text representation model based on a basic model.

[0061] Regarding step S302, please refer to the detailed description in step S202, and details will not be elaborated here.

[0062] S304. Obtain multiple initial sample query statements according to a preset data set corresponding to the target scenario, and multiple initial positive samples and / or multiple initial negative samples and / or multiple initial weak positive samples corresponding to the multiple initial sample query statements.

[0063] Optionally, when constructing samples required for fine-tuning training, positive samples, weakly positive samples, and negative samples can be selected and defined through supervised labels (i.e., quantifiable bases). For example, for an available dataset related to a pre-prepared target scenario, if the data items in the preset dataset include similarity scores between the data, then multiple initial sample query statements can be directly obtained according to the preset dataset, and based on the similarity scores in the dataset, high-score items can be selected as initial positive samples, medium-score items as initial weakly positive samples, and low-score items as initial negative samples. The division of high-score items, medium-score items, and low-score items can be determined according to the actual situation, and the embodiments of this specification do not limit this.

[0064] S306. Perform positive sample amplification operations on the initial sample query statements without initial positive samples; perform negative sample amplification operations on the initial sample query statements without initial negative samples; perform weakly positive sample amplification operations on the initial sample query statements without initial weakly positive samples.

[0065] Furthermore, considering that in a retrieval task, manually annotating existing samples can directly mark a completely correct answer, a partially relevant answer, and a completely wrong answer as available training data. However, in actual operations, it is relatively difficult to find corresponding complete positive, weakly positive, and negative sample pairs for each anchored sample query statement. Therefore, in order to ensure that sample query statements without corresponding comparison samples have corresponding positive, weakly positive, and negative samples, a preset unsupervised sample provider can be added to achieve sample amplification, so as to complete the complement when a certain type of sample is absent, and supplement incomplete sample statement pairs.

[0066] Specifically, when performing sample amplification, different types of samples can be achieved through different unsupervised operations. For example, positive sample amplification operations at least include randomly repeating the original statement, negative sample amplification operations at least include randomly extracting from the dataset, and weakly positive sample amplification operations include at least one of rewriting the syntax tree of the original statement, randomly replacing, randomly inserting, randomly shuffling, and splicing positive and negative samples. Usually during the training process, for each sample query statement, the number of positive example statement pairs and weakly positive example statement pairs required is relatively small, generally 1 - 2 are sufficient, while multiple negative example statement pairs are required. Therefore, the embodiments of this specification do not limit the specific number of positive example statement pairs, weakly positive example statement pairs, and negative example statement pairs for each sample query statement, and can be selected according to actual needs during application. This randomized unsupervised sample amplification method helps to supplement the lacking comparison samples, ensuring the diversity and richness of the training data, and thus enabling the model to learn more comprehensively from the data.

[0067] S308. When the multiple sample statement pairs include positive example statement pairs with the sample comparison statement being a positive sample and negative example statement pairs with the sample comparison statement being a negative sample, the training task at least includes a contrast task; during the training process of the initial text representation model, control the initial text representation model to distinguish the positive samples and negative samples corresponding to the sample query statements in each sample statement pair for the multiple sample statement pairs respectively.

[0068] Optionally, for the fine-tuning training of the initial text representation model, traditional contrast learning can be first performed based on the positive sample pairs and negative sample pairs, enabling the model to distinguish similar samples (positive samples) and dissimilar samples (negative samples), thereby improving the representation ability of the model.

[0069] Specifically, when performing the contrast task, control the initial text representation model to distinguish the positive samples and negative samples corresponding to the sample query statements in each sample statement pair for the multiple positive example statement pairs with the sample comparison statement being a positive sample and negative example statement pairs with the sample comparison statement being a negative sample respectively.

[0070] S310. Calculate the contrast loss value corresponding to the contrast task according to the execution result of the contrast task.

[0071] Furthermore, when the initial text representation model outputs the prediction and discrimination results for the positive samples and negative samples, the contrast loss value corresponding to this contrast task can be calculated. Specifically, for n sample statement pairs, where q represents the sample query statement in each sample statement pair, a + represents the positive sample corresponding to each sample query statement, a - represents the negative sample corresponding to each sample query statement, and s(q, a) represents the similarity score between the sample query statement q and the sample comparison statement a, then the contrast loss function L c can be defined as follows:

[0072]

[0073] Among them, Z1 represents the sum of the scores of all types of negative example statement pairs and their corresponding numerators (which are positive example statement pairs at this time) in the contrast prediction results, and its definition is:

[0074]

[0075] In the above formula, e() is the exponential function used to amplify the similarity score, and τ is the temperature hyperparameter (temperature) used to control the smoothness of the softmax activation function of the model. Through the above formula of the contrast loss, encourage the initial text representation model to accurately distinguish the positive samples and negative samples of the sample query statements.

[0076] S312. When the multiple sample statement pairs include positive example statement pairs with positive sample comparison statements, negative example statement pairs with negative sample comparison statements, and weakly positive example statement pairs with weakly positive sample comparison statements, the training task at least includes a ranking task; control the initial text representation model to rank the positive samples, negative samples, and weakly positive samples corresponding to the sample query statements in each sample statement pair from high to low according to the similarity with the sample query statements.

[0077] Optionally, in addition to contrastive learning, the text representation model can also perform ranking task learning through positive example statement pairs with positive sample comparison statements, negative example statement pairs with negative sample comparison statements, and weakly positive example statement pairs with weakly positive sample comparison statements included in the multiple sample statement pairs, so as to train the accuracy of the similarity scores output by itself through the ranking task. In the ranking task, the initial text representation model will be controlled to rank the positive samples, negative samples, and weakly positive samples corresponding to the sample query statements in each sample statement pair from high to low according to the similarity with the sample query statements.

[0078] Specifically, if represents the correct ranking sequence of a total of m items involved in the comparison (1, 2, 3,..., m represents the correct order), and s represents the actual ranking score, then the probability of the correct sequence column is defined as:

[0079]

[0080] In the embodiments of this specification, in the case of multiple negative samples, the model can rank the positive samples, weakly positive samples, and negative samples of each sample, and when sorting, rank the positive sample of each sample query statement first, the weakly positive sample second, and all negative samples tied for third.

[0081] S314. Calculate the ranking loss value corresponding to the ranking task according to the execution result of the ranking task.

[0082] Furthermore, since the ranking score and order are also determined by the similarity score s(q, a) between the query and the answer, then q still represents the sample query statement in each sample statement pair, a + still represents the positive sample corresponding to each sample query statement, a - still represents the negative sample corresponding to each sample query statement, and in addition, a w+ represents the weakly positive sample corresponding to each sample query statement. The ranking loss function L of the correct sequence probability of the model for a batch of samples l can be expressed as:

[0083]

[0084] Among them, Z2 represents the sum of the scores of all types of negative sample pairs corresponding to the numerators (which are weakly positive example sentence pairs at this time) when used as denominators, and its definition is:

[0085]

[0086] In the above formula, e() is an exponential function used to amplify the similarity score, and τ is the temperature hyperparameter used to control the smoothness of the softmax activation function of the model. Through learning the ranking task, the model will be prompted to have the ability to output similarity scores that reflect the correct ranking for each sample.

[0087] S316. When the multiple sample sentence pairs include positive example sentence pairs with positive sample comparison sentences, negative example sentence pairs with negative sample comparison sentences, and weakly positive example sentence pairs with weakly positive sample comparison sentences, the training task at least includes a classification task; control the initial text representation model to perform label classification on the positive samples, negative samples, and weakly positive samples corresponding to the sample query sentences in each sample sentence pair for the multiple sample sentence pairs.

[0088] On the other hand, it is also possible to make the initial text representation model learn the classification task through the positive example sentence pairs with positive sample comparison sentences, negative example sentence pairs with negative sample comparison sentences, and weakly positive example sentence pairs with weakly positive sample comparison sentences included in the multiple sample sentence pairs, so as to train the classification accuracy of the model for different types of samples through the ranking task. In the classification task, the initial text representation model will be controlled to perform label classification on the positive samples, negative samples, and weakly positive samples corresponding to the sample query sentences in each sample sentence pair for the multiple sample sentence pairs.

[0089] S318. Calculate the classification loss value corresponding to the classification task according to the execution result of the classification task.

[0090] Furthermore, the embodiments of this specification enhance the similarity scores generated by the model through cross-entropy loss, thereby effectively distinguishing three levels of pairings. Then the classification loss function L e is expressed as:

[0091]

[0092] Among them, N(pair) represents the total number of positive samples, weakly positive samples, and negative samples. The variable y p represents the true labels of three different pairing levels, and z p,k represents the classification logits value of sample pair p on sample pair category k. The constant μ e，d is set to 0.1 in the standard configuration.

[0093] It should be noted that in the loss function formulas of the above-mentioned various tasks, the temperature hyperparameter τ is uniformly set to 0.05.

[0094] S320. Select the loss weighting coefficients corresponding to each training task based on ablation experiments; calculate the total loss value of the initial text representation model based on the loss weighting coefficients corresponding to each training task and the task loss values, and adjust the parameters of the initial text representation model based on the total loss value until the initial text representation model converges to obtain the trained text representation model.

[0095] Optionally, after the model obtains the loss values corresponding to several training tasks respectively, it is also necessary to calculate the total loss value according to the task loss values, so as to adjust the parameters of the initial text representation model until the initial text representation model converges.

[0096] Specifically, the loss weighting coefficients corresponding to each task loss value in the final total loss value can be selected through ablation experiments first. The loss weighting coefficients reflect the influence of each task loss value on the final total loss value. First, the calculation formula of the total loss value Loss is expressed as:

[0097] Loss = μ c ·L c + μ l ·L l + μ e ·L e ;

[0098] In a feasible implementation manner, μ c = 2.0, μ l = 1.0, μ e = 0.2 are finally selected as the standard configuration through ablation experiments. The specific numerical values and acquisition methods of the set values of the fixed hyperparameters such as constants and coefficients in the above formula are not uniquely limited in the embodiments of this specification. By unifying the contrast learning, classification, and ranking tasks into a composite loss function in this way, cross-task deep feature learning is realized, and it can be optimized in multiple directions simultaneously in a single framework, greatly improving the learning efficiency of the framework.

[0099] After training convergence in this way, a text representation model that can be applied to actual scenarios to complete specific tasks can be obtained, and specifically, tasks under requirements such as text vector representation, query, and retrieval can be completed.

[0100] In the embodiments of this specification, a method for training a text representation model is provided. Positive, negative, and weakly positive samples are amplified through an unsupervised sample provider to efficiently supplement incomplete sample sentence pairs and ensure the diversity and richness of training data. By means of contrast loss, ranking loss, and classification loss, the contrast, classification, and ranking tasks among text positive samples, weakly positive samples, and negative samples are simultaneously optimized. This loss function enables the model to learn the hierarchical differences among three different levels of sample pairs, thereby achieving a better similarity score distribution in the vector space.

[0101] Please refer to Figure 4 , Figure 4 which is a structural block diagram of a training device for a text representation model provided in the embodiments of this specification. As Figure 4 shown, the training device 400 for the text representation model includes:

[0102] A model construction module 410, configured to construct an initial text representation model based on a basic model;

[0103] A sample preparation module 420, configured to obtain multiple sample sentence pairs related to a target scenario. The sample sentence pair is a sentence pair composed of a sample query sentence and a sample comparison sentence, and the sample comparison sentence is a positive sample and / or a negative sample and / or a weakly positive sample corresponding to the sample query sentence;

[0104] A model training module 430, configured to train the initial text representation model based on multiple sample sentence pairs;

[0105] A model learning module 440, configured to, during the training process of the initial text representation model, control the initial text representation model to execute at least two training tasks for multiple sample sentence pairs, calculate the task loss values corresponding to the respective training tasks according to the execution results of the respective training tasks, and adjust the parameters of the initial text representation model based on the respective task loss values until the initial text representation model converges, thereby obtaining a trained text representation model.

[0106] Optionally, when the multiple sample sentence pairs include positive example sentence pairs with positive samples as sample comparison sentences and negative example sentence pairs with negative samples as sample comparison sentences, the training tasks at least include a contrast task; the training device 400 for the text representation model further includes: a contrast module, configured to, when executing the contrast task, control the initial text representation model to distinguish the positive samples and negative samples corresponding to the sample query sentences in the multiple sample sentence pairs respectively.

[0107] Optionally, when the multiple sample statement pairs include positive example statement pairs where the sample comparison statement is a positive sample, negative example statement pairs where the sample comparison statement is a negative sample, and weakly positive example statement pairs where the sample comparison statement is a weakly positive sample, the training task at least includes a ranking task; the training apparatus 400 for the text representation model further includes: a ranking module, configured to, when executing the ranking task, control the initial text representation model to rank the positive samples, negative samples, and weakly positive samples corresponding to the sample query statement in each sample statement pair in descending order of similarity to the sample query statement.

[0108] Optionally, when the multiple sample statement pairs include positive example statement pairs where the sample comparison statement is a positive sample, negative example statement pairs where the sample comparison statement is a negative sample, and weakly positive example statement pairs where the sample comparison statement is a weakly positive sample, the training task at least includes a classification task; the training apparatus 400 for the text representation model further includes: a classification module, configured to, when executing the classification task, control the initial text representation model to perform label classification on the positive samples, negative samples, and weakly positive samples corresponding to the sample query statement in each sample statement pair.

[0109] Optionally, the training apparatus 400 for the text representation model further includes: a coefficient selection module, configured to select the loss weighting coefficient corresponding to each training task based on an ablation experiment; the model learning module 440 is further configured to calculate the total loss value of the initial text representation model based on the loss weighting coefficient corresponding to each training task and the task loss value, and adjust the parameters of the initial text representation model until the initial text representation model converges.

[0110] Optionally, the sample preparation module 420 is further configured to obtain a plurality of initial sample query statements, and a plurality of initial positive samples and / or a plurality of initial negative samples and / or a plurality of initial weakly positive samples corresponding to the plurality of initial sample query statements according to a preset data set corresponding to the target scenario; perform a positive sample amplification operation on the initial sample query statement without an initial positive sample; perform a negative sample amplification operation on the initial sample query statement without an initial negative sample; perform a weakly positive sample amplification operation on the initial sample query statement without an initial weakly positive sample.

[0111] Optionally, the positive sample amplification operation at least includes random repetition, the negative sample amplification operation at least includes random extraction, and the weakly positive sample amplification operation includes at least one of syntax tree rewriting, random replacement, random insertion, random shuffling, and positive and negative sample splicing.

[0112] Optionally, the positive sample amplification operation and / or the negative sample amplification operation and / or the weakly positive sample amplification operation is implemented based on a preset unsupervised sample provider.

[0113] In the embodiments of this specification, a training device for a text representation model is provided. Among them, a model construction module is used to construct an initial text representation model based on a basic model; a sample preparation module is used to obtain multiple sample statement pairs related to a target scenario, where a sample statement pair is a statement pair composed of a sample query statement and a sample comparison statement, and the sample comparison statement is a positive sample and / or a negative sample and / or a weak positive sample corresponding to the sample query statement; a model training module is used to train the initial text representation model based on multiple sample statement pairs; a model learning module is used to, during the training process of the initial text representation model, control the initial text representation model to execute at least two training tasks for multiple sample statement pairs, calculate the task loss value corresponding to each training task according to the execution result of each training task, and adjust the parameters of the initial text representation model based on each task loss value until the initial text representation model converges, so as to obtain a trained text representation model. In the embodiments of this specification, a contrastive learning mechanism of three levels of positive samples, weak positive samples, and negative samples is adopted to construct a multi-level contrastive learning framework for the model, and the processing ability of the model for complex tasks is optimized through carefully divided sample categories. This hierarchical contrastive relationship prompts the model to deepen its understanding of data in multiple dimensions. Further, this mechanism enables the model to predict multiple different types of output targets during the training process, enabling the model to accurately capture and learn the characteristics of samples in different aspects, and optimizing the learning of relevant characteristics of the model for specific tasks through the hierarchical contrastive relationship, significantly improving the discrimination and robustness of text representation. In the stage of model parameter tuning, the loss values obtained after predicting multiple output targets are integrated into a unified composite loss function, enabling the model to optimize the characteristics in multiple directions under the target scenario in a single framework. This training and learning mechanism combining multiple output targets greatly improves the learning efficiency and generalization performance of the entire framework in a specific scenario / specific task.

[0114] The embodiments of this specification provide a computer program product containing instructions. When the computer program product runs on a computer or a processor, it causes the computer or the processor to execute the steps of the method in any one of the above embodiments.

[0115] The embodiments of this specification also provide a computer storage medium. The computer storage medium can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor to execute the steps of the method in any one of the above embodiments.

[0116] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of a terminal provided by the embodiments of this specification. As Figure 5 shown, the terminal 500 may include: at least one terminal processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0117] Among them, the communication bus 502 is used to realize the connection and communication between these components.

[0118] Among them, the user interface 503 may include a display screen (Display), a camera (Camera), and optionally, the user interface 503 may further include a standard wired interface and a wireless interface.

[0119] Among them, the network interface 504 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0120] Among them, the terminal processor 501 may include one or more processing cores. The terminal processor 501 connects various parts within the entire terminal 500 by using various interfaces and lines, and executes various functions of the terminal 500 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 505, and by calling the data stored in the memory 505. Optionally, the terminal processor 501 may be implemented in at least one of the hardware forms of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The terminal processor 501 may integrate one or several combinations of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, the user interface, and application programs, etc.; the GPU is responsible for the rendering and drawing of the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the terminal processor 501 and may be implemented separately by a single chip.

[0121] Among them, the memory 505 may include a Random Access Memory (RAM), or may also include a Read-Only Memory (ROM). Optionally, the memory 505 includes a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, code, code sets or instruction sets. The memory 505 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments. Optionally, the memory 505 may also be at least one storage device located far from the aforementioned terminal processor 501. As Figure 5 shown, the memory 505, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a training program for the text representation model.

[0122] In Figure 5 the terminal 500 shown, the user interface 503 is mainly used to provide an input interface for the user to obtain user input data; while the terminal processor 501 can be used to call the training program of the text representation model stored in the memory 505 and specifically perform the following operations:

[0123] Construct an initial text representation model based on the base model;

[0124] Obtain multiple sample statement pairs related to the target scenario. The sample statement pair is a statement pair composed of a sample query statement and a sample comparison statement. The sample comparison statement is a positive sample and / or negative sample and / or weak positive sample corresponding to the sample query statement;

[0125] Train the initial text representation model based on multiple sample statement pairs;

[0126] During the training process of the initial text representation model, control the initial text representation model to execute at least two training tasks for multiple sample statement pairs, calculate the task loss value corresponding to each training task according to the execution results of each training task, and adjust the parameters of the initial text representation model based on each task loss value until the initial text representation model converges to obtain the trained text representation model.

[0127] In some embodiments, when the multiple sample statement pairs include positive example statement pairs with a sample comparison statement as a positive sample and negative example statement pairs with a sample comparison statement as a negative sample, the training task at least includes a contrast task; the terminal processor 501 further specifically performs the following steps: when performing the contrast task, control the initial text representation model to distinguish the positive samples and negative samples corresponding to the sample query statements in each of the multiple sample statement pairs respectively.

[0128] In some embodiments, when the multiple sample statement pairs include positive example statement pairs with a sample comparison statement as a positive sample, negative example statement pairs with a sample comparison statement as a negative sample, and weak positive example statement pairs with a sample comparison statement as a weak positive sample, the training task at least includes a ranking task; the terminal processor 501 further specifically performs the following steps: when performing the ranking task, control the initial text representation model to rank the positive samples, negative samples, and weak positive samples corresponding to the sample query statements in each of the multiple sample statement pairs in descending order of similarity to the sample query statements.

[0129] In some embodiments, when the multiple sample statement pairs include positive example statement pairs with a sample comparison statement as a positive sample, negative example statement pairs with a sample comparison statement as a negative sample, and weak positive example statement pairs with a sample comparison statement as a weak positive sample, the training task at least includes a classification task; the terminal processor 501 further specifically performs the following steps: when performing the classification task, control the initial text representation model to perform label classification on the positive samples, negative samples, and weak positive samples corresponding to the sample query statements in each of the multiple sample statement pairs.

[0130] In some embodiments, the terminal processor 501 further specifically performs the following steps: select the loss weighting coefficients corresponding to each training task based on ablation experiments; when the terminal processor 501 adjusts the parameters of the initial text representation model based on the loss values of each task until the initial text representation model converges, it specifically performs the following steps: calculate the total loss value of the initial text representation model based on the loss weighting coefficients corresponding to each training task and the task loss values, and adjust the parameters of the initial text representation model based on the total loss value until the initial text representation model converges.

[0131] In some embodiments, when the terminal processor 501 performs the operation of obtaining multiple sample statement pairs related to the target scenario, it specifically performs the following steps: obtain multiple initial sample query statements according to the preset data set corresponding to the target scenario, and multiple initial positive samples and / or multiple initial negative samples and / or multiple initial weak positive samples corresponding to the multiple initial sample query statements; perform positive sample amplification operations on the initial sample query statements without initial positive samples; perform negative sample amplification operations on the initial sample query statements without initial negative samples; perform weak positive sample amplification operations on the initial sample query statements without initial weak positive samples.

[0132] In some embodiments, the positive sample amplification operation includes at least random duplication, the negative sample amplification operation includes at least random extraction, and the weak positive sample amplification operation includes at least one of syntax tree rewriting, random replacement, random insertion, random scrambling, and positive and negative sample splicing.

[0133] In some embodiments, the positive sample amplification operation and / or the negative sample amplification operation and / or the weak positive sample amplification operation are implemented based on a preset unsupervised sample provider.

[0134] In several embodiments provided in this specification, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or modules can be in an electrical, mechanical or other form.

[0135] The modules described as separate components may or may not be physically separated. The components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, the processes or functions described above in accordance with the embodiments of this specification are generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions can be stored in a computer-readable storage medium or transmitted through the above computer-readable storage medium. The above computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The above computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The above available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a Digital Versatile Disc (DVD)), or a semiconductor medium (for example, a Solid State Disk (SSD)), etc.

[0137] It should be noted that, for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of this specification are not limited by the described order of actions, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential for the embodiments of this specification.

[0138] In addition, it should also be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions. For example, the sample sentence pair data involved in this specification are all obtained under full authorization.

[0139] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0141] The above is the description of a training method, device, storage medium, and terminal for a text representation model provided by the embodiments of this specification. For those skilled in the art, based on the ideas of the embodiments of this specification, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on the embodiments of this specification.

Claims

1. A training method for a text representation model, the method comprising: Constructing an initial text representation model based on a base model; Obtaining a plurality of sample statement pairs related to a target scenario, the sample statement pairs being statement pairs composed of a sample query statement and a sample comparison statement, and the sample comparison statement being a positive sample and / or a negative sample and / or a weak positive sample corresponding to the sample query statement; Training the initial text representation model based on the plurality of sample statement pairs; During the training process of the initial text representation model, controlling the initial text representation model to execute at least two training tasks for the plurality of sample statement pairs, calculating task loss values corresponding to the respective training tasks according to the execution results of the respective training tasks, and adjusting the parameters of the initial text representation model based on the respective task loss values until the initial text representation model converges, thereby obtaining a trained text representation model.

2. The method according to claim 1, when the plurality of sample statement pairs include a positive example statement pair in which the sample comparison statement is the positive sample and a negative example statement pair in which the sample comparison statement is the negative sample, the training task at least includes a contrast task; The method further comprises: When executing the contrast task, controlling the initial text representation model to distinguish the positive sample and the negative sample corresponding to the sample query statement in each of the plurality of sample statement pairs.

3. The method according to claim 1, when the plurality of sample statement pairs include a positive example statement pair in which the sample comparison statement is the positive sample, a negative example statement pair in which the sample comparison statement is the negative sample, and a weak positive example statement pair in which the sample comparison statement is the weak positive sample, the training task at least includes a ranking task; The method further comprises: When executing the ranking task, controlling the initial text representation model to rank the positive sample, the negative sample, and the weak positive sample corresponding to the sample query statement in each of the plurality of sample statement pairs in descending order of similarity to the sample query statement.

4. The method according to claim 1, when the plurality of sample statement pairs include a positive example statement pair in which the sample comparison statement is the positive sample, a negative example statement pair in which the sample comparison statement is the negative sample, and a weak positive example statement pair in which the sample comparison statement is the weak positive sample, the training task at least includes a classification task; The method further comprises: When executing the classification task, controlling the initial text representation model to perform label classification on the positive sample, the negative sample, and the weak positive sample corresponding to the sample query statement in each of the plurality of sample statement pairs.

5. The method according to claim 1, the method further comprises: Selecting loss weighting coefficients corresponding to the respective training tasks based on ablation experiments; The adjusting the parameters of the initial text representation model based on the respective task loss values until the initial text representation model converges includes: Calculate the total loss value of the initial text representation model based on the loss weighting coefficients corresponding to each training task and the task loss values, and adjust the parameters of the initial text representation model based on the total loss value until the initial text representation model converges.

6. The method according to claim 1, wherein the obtaining of the multiple sample statement pairs related to the target scenario includes: Obtaining a plurality of initial sample query statements according to a preset data set corresponding to the target scenario, and a plurality of initial positive samples and / or a plurality of initial negative samples and / or a plurality of initial weak positive samples corresponding to the plurality of initial sample query statements; Performing a positive sample amplification operation on an initial sample query statement without an initial positive sample; Performing a negative sample amplification operation on an initial sample query statement without an initial negative sample; Performing a weak positive sample amplification operation on an initial sample query statement without an initial weak positive sample.

7. The method according to claim 6, wherein the positive sample amplification operation at least includes random repetition, the negative sample amplification operation at least includes random extraction, and the weak positive sample amplification operation includes at least one of syntax tree rewriting, random replacement, random insertion, random shuffling, and positive and negative sample splicing.

8. The method according to claim 6, wherein the positive sample amplification operation and / or the negative sample amplification operation and / or the weak positive sample amplification operation are implemented based on a preset unsupervised sample provider.

9. A training device for a text representation model, the device comprising: A model construction module, configured to construct an initial text representation model based on a basic model; A sample preparation module, configured to obtain a plurality of sample statement pairs related to the target scenario, the sample statement pairs being statement pairs composed of a sample query statement and a sample comparison statement, and the sample comparison statement being a positive sample and / or a negative sample and / or a weak positive sample corresponding to the sample query statement; A model training module, configured to train the initial text representation model based on the plurality of sample statement pairs; A model learning module, configured to, during the training process of the initial text representation model, control the initial text representation model to execute at least two training tasks for the plurality of sample statement pairs, calculate the task loss values corresponding to the respective training tasks according to the execution results of the respective training tasks, and adjust the parameters of the initial text representation model based on the respective task loss values until the initial text representation model converges, to obtain a trained text representation model.

10. A computer program product containing instructions, which, when the computer program product runs on a computer or a processor, causes the computer or the processor to execute the steps of the method according to any one of claims 1 to 8.

11. A computer storage medium, the computer storage medium storing a plurality of instructions, the instructions being adapted to be loaded and executed by a processor to execute the steps of the method according to any one of claims 1 to 8.

12. A terminal, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

Images