Text contrast learning method and device
By using contrast learning methods and specific loss functions in text comparison learning, the existing methods are solved, and more efficient learning and recognition capabilities are achieved.
Patent Information
- Application Number
- CN202411955427.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-28
- Publication Date
- 2025-05-02
Smart Images
Figure CN119917867A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing, and in particular to a text comparison learning method and device. Background Art
[0002] Currently, many text contrastive learning methods are fine-tuned based on pre-trained language models (such as BERT, GPT-3, etc.). For example, when using BERT to calculate text similarity, text pairs are usually taken as input, feature representations are extracted through the pre-trained BERT model, and then the similarity between feature representations is calculated to judge the similarity of the texts. Although existing text contrastive learning methods have achieved certain success in many application scenarios, there are still some significant defects and deficiencies, which are mainly reflected in the following aspects:
[0003] (1) Strong data dependence
[0004] Existing methods usually rely on a large amount of high-quality annotated data. However, in many practical application scenarios, obtaining a large amount of annotated data is both expensive and time-consuming. Even with a large amount of data, the problem of data imbalance still exists, which may cause the model to perform poorly on specific categories. In addition, generating high-quality training data is also challenging, especially the need to cover a variety of possible semantic variations and contexts.
[0005] (2) Insufficient model generalization ability
[0006] Although pre-trained language models (such as BERT, GPT-3) perform well in many NLP tasks, their generalization ability is still insufficient in text contrastive learning tasks. In particular, when applied to unseen domains or new tasks, the performance of the model may drop significantly. This is because the semantic representation ability of the pre-trained model may be insufficient in certain specific domains or tasks, resulting in inaccurate judgment of text similarity.
[0007] (3) Low computational efficiency
[0008] In practical applications, the efficiency of text similarity calculation is particularly important, especially in systems that require real-time response. Traditional text contrastive learning methods, especially those based on the twin-tower model and triplet loss, usually require the calculation of a large number of feature representations and similarities, which has a high computational cost and affects the system's response speed and user experience.
[0009] (4) Lack of fine-grained similarity measurement
[0010] Existing methods usually only consider the overall similarity of text pairs, while ignoring the more fine-grained similarity measurement within the text. For example, although two texts are similar overall, they may have significant differences in some local details, which may be very important for specific application scenarios. Therefore, existing methods are insufficient in dealing with fine-grained similarity measurement. Summary of the invention
[0011] In view of the problems existing in the background technology, a text contrast learning method and device are proposed. First, the present invention adopts a contrast learning method, and by designing a specific loss function, the model can better distinguish the relationship between the query, positive samples and negative samples during the training process. Secondly, in the model reasoning stage, the present invention calculates the similarity between the query and each candidate sample, and selects the sample with the highest similarity as the final result. This method has good application prospects in many text comparison tasks, such as text similarity calculation, text classification, etc.
[0012] The present invention proposes a text contrast learning method, the steps comprising:
[0013] S1. Model training stage
[0014] S11, input processing: input a query, a positive sample and a negative sample; input the query, positive sample and negative sample into the encoder respectively to obtain their feature representation vectors Q, P and N;
[0015] S12, feature fusion: concatenate Q and P, Q and N, P and N respectively to obtain new feature representations QP, QN and PN; process the concatenated feature representations through the fully connected layer to obtain the corresponding output vector;
[0016] S13, loss calculation: Calculate the loss value (loss_1) between the query and the positive sample to make it as close as possible; calculate the loss value (loss_2) between the query and the negative sample to make it as close to 1 as possible; calculate the loss value (loss_3) between the positive sample and the negative sample to make it as close to 1 as possible;
[0017] S14, total loss function: The total loss function is designed to be L = αloss_1 + βloss_2 + γ*loss_3, where α, β and γ are weight parameters of the loss function, which are optimized and adjusted during the training process; by optimizing the total loss function, the model can better distinguish the relationship between query, positive samples and negative samples during the training process;
[0018] S2. Model inference stage
[0019] S21, input processing: In practical applications, a query and multiple candidate samples are input;
[0020] S22, feature representation generation: input the query and candidate samples into the encoder respectively to obtain their feature representation vectors;
[0021] S23, feature fusion: concatenate the feature representations of the query and each candidate sample, process them through a fully connected layer, and obtain the corresponding output vector;
[0022] S24, similarity calculation: calculate the similarity between the query and each candidate sample, and select the sample with the highest similarity as the final result.
[0023] Preferably, in S11, a pre-trained language model is selected by an encoder for fine-tuning to adapt to a specific text comparison task.
[0024] Preferably, the selected pre-trained language model is BERT or GPT-4.
[0025] Preferably, the multiple candidate samples in S21 include positive samples and negative samples.
[0026] The present invention further proposes a text contrast learning device for implementing the above text contrast learning method, comprising a model training module and a model reasoning module;
[0027] The model training module includes:
[0028] Input processing unit, used for inputting query, positive sample and negative sample; by inputting query, positive sample and negative sample into encoder, to obtain their feature representation vectors Q, P and N;
[0029] The feature fusion unit is used to concatenate Q and P, Q and N, and P and N respectively to obtain new feature representations QP, QN, and PN; the concatenated feature representations are processed through the fully connected layer to obtain the corresponding output vector;
[0030] The loss calculation unit is used to calculate the loss value between the query and the positive sample, the loss value between the query and the negative sample, and the loss value between the positive sample and the negative sample;
[0031] and a total loss function unit, used to design a total loss function;
[0032] The model inference module includes:
[0033] Input processing unit, used to input a query and multiple candidate samples;
[0034] The feature representation generation unit is used to input the query and candidate samples into the encoder to obtain their feature representation vectors;
[0035] The feature fusion unit is used to concatenate the feature representations of the query and each candidate sample, and process them through the fully connected layer to obtain the corresponding output vector;
[0036] And the similarity calculation unit is used to calculate the similarity between the query and each candidate sample, and select the sample with the highest similarity as the final result
[0037] Compared with the prior art, the present invention has the following beneficial technical effects:
[0038] (1) Efficient learning strategy: The present invention adopts a contrastive learning method to effectively improve the learning efficiency and recognition accuracy of the model by comparing positive samples and negative samples.
[0039] (2) Powerful feature extraction capability: The present invention converts the input query and sample into a feature representation vector through an encoder, which can effectively extract the key features of the input information.
[0040] (3) Flexible loss function design: The present invention designs a total loss function. By weighting different loss values, the model's attention to different types of errors can be flexibly adjusted, so that the model can better distinguish the relationship between queries, positive samples, and negative samples during training. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a flow chart of the text contrast learning method of the present invention. DETAILED DESCRIPTION
[0042] Embodiment 1
[0043] like Figure 1 As shown, a text contrast learning method proposed by the present invention comprises the following steps:
[0044] S1. Model training stage
[0045] S11, input processing: input a query, a positive sample and a negative sample; input the query, positive sample and negative sample into the encoder respectively to obtain their feature representation vectors Q, P and N; select the pre-trained language model through the encoder for fine-tuning to adapt to the specific text comparison task. The selected pre-trained language models are BERT and GPT-4;
[0046] S12, feature fusion: concatenate Q and P, Q and N, P and N respectively to obtain new feature representations QP, QN and PN; process the concatenated feature representations through the fully connected layer to obtain the corresponding output vector;
[0047] S13, loss calculation: Calculate the loss value (loss_1) between the query and the positive sample to make it as close as possible; calculate the loss value (loss_2) between the query and the negative sample to make it as close to 1 as possible; calculate the loss value (loss_3) between the positive sample and the negative sample to make it as close to 1 as possible;
[0048] S14, total loss function: The total loss function is designed to be L = αloss_1 + βloss_2 + γ*loss_3, where α, β and γ are weight parameters of the loss function, which are optimized and adjusted during the training process; by optimizing the total loss function, the model can better distinguish the relationship between query, positive samples and negative samples during the training process;
[0049] S2. Model inference stage
[0050] S21, input processing: In practical applications, a query and multiple candidate samples are input; the multiple candidate samples include positive samples and negative samples;
[0051] S22, feature representation generation: input the query and candidate samples into the encoder respectively to obtain their feature representation vectors;
[0052] S23, feature fusion: concatenate the feature representations of the query and each candidate sample, process them through a fully connected layer, and obtain the corresponding output vector;
[0053] S24, similarity calculation: calculate the similarity between the query and each candidate sample, and select the sample with the highest similarity as the final result.
[0054] Embodiment 2
[0055] The present invention further proposes a text contrast learning device for implementing the above text contrast learning method, comprising a model training module and a model reasoning module;
[0056] The model training module includes:
[0057] Input processing unit, used for inputting query, positive sample and negative sample; by inputting query, positive sample and negative sample into encoder, to obtain their feature representation vectors Q, P and N;
[0058] The feature fusion unit is used to concatenate Q and P, Q and N, and P and N respectively to obtain new feature representations QP, QN, and PN; the concatenated feature representations are processed through the fully connected layer to obtain the corresponding output vector;
[0059] The loss calculation unit is used to calculate the loss value between the query and the positive sample, the loss value between the query and the negative sample, and the loss value between the positive sample and the negative sample;
[0060] and a total loss function unit, used to design a total loss function;
[0061] The model inference module includes:
[0062] Input processing unit, used to input a query and multiple candidate samples;
[0063] The feature representation generation unit is used to input the query and candidate samples into the encoder to obtain their feature representation vectors;
[0064] The feature fusion unit is used to concatenate the feature representations of the query and each candidate sample, and process them through the fully connected layer to obtain the corresponding output vector;
[0065] And the similarity calculation unit is used to calculate the similarity between the query and each candidate sample, and select the sample with the highest similarity as the final result.
[0066] The present invention designs a specific model structure and loss function so that the model can better distinguish the relationship between query, positive sample and negative sample during the training process, thereby improving the distinguishing ability of the model. In the model reasoning stage, by calculating the similarity between the query and each candidate sample, the sample with the highest similarity is selected as the final result. This method greatly improves the reasoning efficiency of the model. It is found in the experiment that compared with the traditional text comparison method, the method of the present invention has better performance in various text comparison tasks, whether in the distinguishing ability of the model, reasoning efficiency, or in the generalization ability of the model, there is a significant improvement.
[0067] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.
Claims
1. A text contrast learning method, characterized in that the steps include: S1. Model training stage S11, input processing: input a query, a positive sample and a negative sample; Input the query, positive sample and negative sample into the encoder respectively to obtain their feature representation vectors Q, P and N; S12, feature fusion: concatenate Q and P, Q and N, P and N respectively to obtain new feature representations QP, QN and PN; process the concatenated feature representations through the fully connected layer to obtain the corresponding output vector; S13, loss calculation: Calculate the loss value (loss_1) between the query and the positive sample to make it as close as possible; calculate the loss value (loss_2) between the query and the negative sample to make it as close to 1 as possible; Calculate the loss value (loss_3) between positive and negative samples to make it as close to 1 as possible; S14, total loss function: The total loss function is designed to be L = αloss_1 + βloss_2 + γ*loss_3, where α, β and γ are weight parameters of the loss function, which are optimized and adjusted during the training process; by optimizing the total loss function, the model can better distinguish the relationship between query, positive samples and negative samples during the training process; S2. Model inference stage S21, input processing: In practical applications, a query and multiple candidate samples are input; S22, feature representation generation: input the query and candidate samples into the encoder respectively to obtain their feature representation vectors; S23, feature fusion: concatenate the feature representations of the query and each candidate sample, process them through a fully connected layer, and obtain the corresponding output vector; S24, similarity calculation: calculate the similarity between the query and each candidate sample, and select the sample with the highest similarity as the final result.
2. The text contrast learning method according to claim 1, characterized in that: In S11, the encoder selects a pre-trained language model for fine-tuning to adapt to the specific text comparison task.
3. The text contrast learning method according to claim 2, characterized in that: The selected pre-trained language models are BERT and GPT-4.
4. The text contrast learning method according to claim 1, characterized in that: The multiple candidate samples in S21 include positive samples and negative samples.
5. A text contrast learning device, characterized in that: Used to implement the text contrast learning method in claim 1, including a model training module and a model reasoning module; The model training module includes: Input processing unit, used for inputting query, positive sample and negative sample; by inputting query, positive sample and negative sample into encoder, to obtain their feature representation vectors Q, P and N; The feature fusion unit is used to concatenate Q and P, Q and N, and P and N respectively to obtain new feature representations QP, QN, and PN; the concatenated feature representations are processed through the fully connected layer to obtain the corresponding output vector; The loss calculation unit is used to calculate the loss value between the query and the positive sample, the loss value between the query and the negative sample, and the loss value between the positive sample and the negative sample; and a total loss function unit, used to design a total loss function; The model inference module includes: Input processing unit, used to input a query and multiple candidate samples; The feature representation generation unit is used to input the query and candidate samples into the encoder to obtain their feature representation vectors; The feature fusion unit is used to concatenate the feature representations of the query and each candidate sample, and process them through the fully connected layer to obtain the corresponding output vector; And a similarity calculation unit is used to calculate the similarity between the query and each candidate sample, and select the sample with the highest similarity as the final result.