Contrastive learning method and system based on text sensitivity and semantic invariance

Abstract

The present invention provides a contrastive learning method, system, storage medium and electronic device based on text sensitivity and semantic invariance, and relates to the field of contrastive learning technology. Compared with existing methods, the present invention, on the one hand, ensures that the semantics of the input sentence remains unchanged after data enhancement by performing data enhancement at the representation level, thereby more effectively supporting contrastive learning for feature-insensitive learning. On the other hand, the present invention makes the model more targeted in the process of feature-sensitive learning by designing task-related or data-related sensitive feature selection methods, thereby improving the adaptability of the model in downstream tasks, and the sensitive feature selection method is decoupled from the sensitive feature learning process, thereby ensuring that the model can design different sensitive feature selection methods according to specific needs, thereby improving the generalization ability and applicability of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of contrastive learning, and in particular to a contrastive learning method, system, storage medium and electronic device based on text sensitivity and semantic invariance. Background Art

[0002] Contrastive learning is a self-supervised learning method. For anchor samples, positive samples are obtained through data enhancement methods, and negative samples are obtained through random sampling methods. Then, the distance between sample pairs is optimized, the distance between positive sample pairs is reduced, and the distance between negative sample pairs is increased, thereby achieving accurate representation of the input data (image, text).

[0003] Currently, commonly used contrastive learning methods mainly include contrastive learning methods for computer vision and methods for natural language processing. However, for feature-insensitive learning, while the dropout mechanism of the pre-trained model can ensure that the semantics remain unchanged after enhancement, the enhanced samples are not perturbed strongly, resulting in a small difference between the enhanced input sentence representation and the original representation, which prevents the full effectiveness of contrastive learning. In addition, for feature-sensitive learning, on the one hand, using generative methods cannot control the quality of generation, and on the other hand, the feature sensitivity is more focused on textual changes in the words of the input sentence, which may conflict with feature-insensitive learning. At the same time, this feature sensitivity is difficult to transfer to specific downstream tasks. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a contrastive learning method, system, storage medium and electronic device based on text sensitivity and semantic invariance, which solves the technical problem of being unable to accurately represent sample data.

[0006] (2) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A contrastive learning method based on text sensitivity and semantic invariance, including:

[0009] S1. Determine anchor samples and negative samples in the current training batch, and use a pre-trained model to obtain semantic representation matrices of the anchor samples and negative samples, respectively; perform data augmentation on the semantic representation matrices of the anchor samples to obtain a semantically invariant feature-insensitive enhanced representation, and use it as the semantic representation of the positive sample;

[0010] S2. Based on the weight of each word in the anchor sample in the entire training set and the semantic representation matrix of the anchor sample, the pre-trained model is used to obtain a data-related feature-sensitive enhanced representation;

[0011] S3. Obtain the semantic representation matrices of the anchor sample, negative sample, and positive sample, and the semantic representation vectors corresponding to the feature-sensitive enhanced representation, respectively; map the semantic representation vectors corresponding to the semantic representation matrices of the anchor sample, negative sample, and positive sample to different metric spaces to obtain a first mapping output result; map the semantic representation vector corresponding to the feature-sensitive enhanced representation to a final decision space to obtain a second mapping output result;

[0012] S4. Construct a first loss function based on the first mapping output result; construct a second loss function based on the second mapping output result; fuse the first loss function and the second loss function to obtain a final loss function and use it to optimize parameters until the model converges.

[0013] Preferably, in S1:

[0014] Select other non-anchor samples in the same training batch as negative samples of the anchor samples;

[0015] Define the input sentence corresponding to the anchor sample as S i ={s i1 ,s i2 ,…,s il ,…,s iL}, where s il represents the one-hot representation of the lth word in the i-th sentence, where L represents the length of the sentence;

[0016] The pre-trained model is used to obtain the semantic representation matrix of the anchor point sample:

[0017] H i ={h i1 ,h i2 ,…,h il ,…,h iL}=f(S i )

[0018] Among them, H i represents the representation matrix of the i-th sentence after encoding by the pre-training model, where h ij represents the vector representation of the jth word; f() represents the pre-training model;

[0019] Use the same pre-training model to obtain the semantic representation matrix of the negative sample

[0020] Preferably, in said S1: by performing the semantic representation matrix H i The order is disrupted, some feature columns are masked, and the elements in the entire matrix representation are randomly discarded to obtain the feature-insensitive enhanced representation. The process can be expressed as follows:

[0021] op1=random(shuffling,featurecutoff,dropout),

[0022]

[0023] Among them, op1 represents the first data enhancement method, shuffling represents the order shuffling operation, featurecutoff represents the operation of covering part of the feature column, and dropout represents the random dropping operation; It means the semantically invariant feature-insensitive enhanced representation after data enhancement.

[0024] Preferably, the S2 includes:

[0025] S21, using the TF-IDF method to statistically analyze all sentences in the training set, and obtain the weight α of each word in the anchor sample in the entire training set. ij , and compare it with the preset weight threshold θ. If α ij >θ, perform data enhancement on the word corresponding to the weight;

[0026] S22. Combined with the semantic representation matrix of the anchor sample, data enhancement is performed on the words that require data enhancement as determined in the above step with a certain probability β at the word level using WordNet, and the pre-trained model is used to obtain data-related feature-sensitive enhanced representations. The process can be expressed as follows:

[0027] op2=random(Synonym,Delete,Swap,Noop)

[0028] Among them, op2 represents the second data augmentation method, Synonym represents the synonym replacement operation, Delete represents the random deletion operation, Swap represents the random swap operation, and Noop represents no operation;

[0029]

[0030]

[0031]

[0032] in, Represents the one-hot representation of the lth word in the i-th sentence after data enhancement; Indicates the sentence representation after data enhancement; represents the data-dependent feature-sensitive enhanced representation; f() represents the pre-trained model.

[0033] Preferably, the S3 includes:

[0034] S31, using pooling operations to obtain the semantic representation vectors v corresponding to the semantic representation matrices of the anchor sample, negative sample and positive sample respectively i 、

[0035]

[0036] Among them, mean represents average pooling;

[0037] The semantic representation vectors corresponding to the semantic representation matrices of the anchor samples, negative samples and positive samples are mapped to different metric spaces by a first multi-layer perceptron to obtain a first mapping output result z i 、

[0038]

[0039] Among them, MLP1 represents the first multi-layer perceptron;

[0040] S32, using Bi-LSTM to obtain the semantic representation vector corresponding to the feature-sensitive enhanced representation The initial hidden state of the recurrent neural network uses the semantic representation vector v corresponding to the original anchor point sample i ,

[0041]

[0042] The semantic representation vector corresponding to the feature-sensitive enhanced representation is mapped to the final decision space by a second multi-layer perceptron to obtain a second mapping output result.

[0043]

[0044] Among them, MLP2 represents the second multi-layer perceptron.

[0045] Preferably, in S4:

[0046] The first loss function is expressed as:

[0047]

[0048] Where log represents the logarithmic function; sim represents the similarity function; K represents the number of negative samples; τ is the temperature coefficient in the first loss function, which is a hyperparameter;

[0049] The second loss function is expressed as:

[0050]

[0051] Among them, y′ it and They represent the true label and predicted label of the lth word in the i-th enhanced sentence respectively. The labels are divided into the following four categories: task-related words, modified; task-related words, not modified; task-irrelevant words, modified; task-irrelevant words, not modified.

[0052] The final loss function is expressed as:

[0053] L=μL1+(1-μ)L2

[0054] Here, μ is a hyperparameter.

[0055] Preferably, the pre-trained model is a pre-trained BERT network.

[0056] A contrastive learning system based on text sensitivity and semantic invariance, including:

[0057] The first enhancement module is configured to execute S1, determine anchor samples and negative samples in the current training batch, and use a pre-trained model to obtain semantic representation matrices of the anchor samples and negative samples, respectively; perform data enhancement on the semantic representation matrix of the anchor samples to obtain a semantically invariant feature-insensitive enhanced representation, and use it as the semantic representation of the positive sample;

[0058] A second enhancement module is configured to execute S2, and obtain a data-related feature-sensitive enhanced representation using the pre-trained model according to the weight of each word in the anchor sample in the entire training set and the semantic representation matrix of the anchor sample;

[0059] A mapping module is configured to execute S3, obtain the semantic representation matrices of the anchor sample, negative sample, and positive sample, and the semantic representation vectors corresponding to the feature-sensitive enhanced representation, respectively; map the semantic representation vectors corresponding to the semantic representation matrices of the anchor sample, negative sample, and positive sample to different metric spaces to obtain a first mapping output result; and map the semantic representation vector corresponding to the feature-sensitive enhanced representation to a final decision space to obtain a second mapping output result;

[0060] An optimization module is used to execute S4, construct a first loss function based on the first mapping output result; construct a second loss function based on the second mapping output result; and fuse the first loss function and the second loss function to obtain a final loss function and use it to optimize parameters until the model converges.

[0061] A storage medium stores a computer program for contrastive learning based on text sensitivity and semantic invariance, wherein the computer program enables a computer to execute the contrastive learning method described above.

[0062] An electronic device, comprising:

[0063] one or more processors;

[0064] Memory; and

[0065] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for executing the contrastive learning method as described above.

[0066] (3) Beneficial effects

[0067] The present invention provides a comparative learning method, system, storage medium, and electronic device based on text sensitivity and semantic invariance. Compared with the existing technology, it has the following advantages:

[0068] In the present invention, on the one hand, data enhancement is performed on the semantic representation matrix of the anchor sample to obtain a semantically unchanged feature-insensitive enhanced representation, and use it as the semantic representation of the positive sample; that is, by performing data enhancement at the representation level, it is ensured that the semantics of the input sentence remains unchanged after data enhancement, thereby more effectively supporting contrastive learning for feature-insensitive learning. On the other hand, based on the weight of each word in the anchor sample in the entire training set and the semantic representation matrix of the anchor sample, a pre-training model is used to obtain data-related feature-sensitive enhanced representation; that is, by designing a task-related or data-related sensitive feature selection method, the model is made more targeted in the feature-sensitive learning process, thereby improving the model's adaptability in downstream tasks, and the sensitive feature selection method is decoupled from the sensitive feature learning process, thereby ensuring that the model can design different sensitive feature selection methods according to specific needs, thereby improving the model's generalization ability and applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 A flowchart of a contrastive learning method based on text sensitivity and semantic invariance provided by an embodiment of the present invention;

[0071] Figure 2 This is a structural block diagram of a contrastive learning system based on text sensitivity and semantic invariance provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0073] The embodiments of the present application solve the technical problem of being unable to accurately represent sample data by providing a contrastive learning method, system, storage medium and electronic device based on text sensitivity and semantic invariance.

[0074] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0075] In an embodiment of the present invention, on the one hand, data enhancement is performed on the semantic representation matrix of the anchor sample to obtain a semantically unchanged feature-insensitive enhanced representation, and use it as the semantic representation of the positive sample; that is, by performing data enhancement at the representation level, it is ensured that the semantics of the input sentence remains unchanged after data enhancement, thereby more effectively supporting contrastive learning for feature-insensitive learning. On the other hand, based on the weight of each word in the anchor sample in the entire training set and the semantic representation matrix of the anchor sample, a pre-training model is used to obtain data-related feature-sensitive enhanced representation; that is, by designing a task-related or data-related sensitive feature selection method, the model is made more targeted in the feature-sensitive learning process, thereby improving the model's adaptability to downstream tasks, and the sensitive feature selection method is decoupled from the sensitive feature learning process, thereby ensuring that the model can design different sensitive feature selection methods according to specific needs, thereby improving the model's generalization ability and applicability.

[0076] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0077] Example:

[0078] like Figure 1 As shown, an embodiment of the present invention provides a contrastive learning method based on text sensitivity and semantic invariance, including:

[0079] S1. Determine anchor samples and negative samples in the current training batch, and use a pre-trained model to obtain semantic representation matrices of the anchor samples and negative samples, respectively; perform data augmentation on the semantic representation matrices of the anchor samples to obtain a semantically invariant feature-insensitive enhanced representation, and use it as the semantic representation of the positive sample;

[0080] S2. Based on the weight of each word in the anchor sample in the entire training set and the semantic representation matrix of the anchor sample, the pre-trained model is used to obtain a data-related feature-sensitive enhanced representation;

[0081] S3. Obtain the semantic representation matrices of the anchor sample, negative sample, and positive sample, and the semantic representation vectors corresponding to the feature-sensitive enhanced representation, respectively; map the semantic representation vectors corresponding to the semantic representation matrices of the anchor sample, negative sample, and positive sample to different metric spaces to obtain a first mapping output result; map the semantic representation vector corresponding to the feature-sensitive enhanced representation to a final decision space to obtain a second mapping output result;

[0082] S4. Construct a first loss function based on the first mapping output result; construct a second loss function based on the second mapping output result; fuse the first loss function and the second loss function to obtain a final loss function and use it to optimize parameters until the model converges.

[0083] Compared with existing methods, the embodiments of the present invention, on the one hand, ensure that the semantics of the input sentence remains unchanged after data enhancement by performing data enhancement at the representation level, thereby more effectively supporting contrastive learning for feature-insensitive learning. On the other hand, by designing task-related or data-related sensitive feature selection methods, the embodiments of the present invention make the model more targeted during feature-sensitive learning, improving the model's adaptability to downstream tasks. Moreover, the sensitive feature selection method is decoupled from the sensitive feature learning process, ensuring that the model can design different sensitive feature selection methods according to specific needs, thereby improving the model's generalization ability and applicability.

[0084] The following will introduce each step of the above technical solution in detail with specific content:

[0085] In step S1, anchor samples and negative samples are determined in the current training batch, and the semantic representation matrices of the anchor samples and negative samples are respectively obtained using a pre-training model; data enhancement is performed on the semantic representation matrix of the anchor samples to obtain a semantically invariant feature-insensitive enhanced representation, which is used as the semantic representation of the positive sample.

[0086] In this step, other non-anchor samples in the same training batch are selected as negative samples of the anchor samples; the input sentence corresponding to the anchor sample is defined as S i ={s i1 ,s i2 ,…,s il ,…,s iL}, where s il represents the one-hot representation of the lth word in the i-th sentence, and L represents the length of the sentence.

[0087] The pre-trained model is used to obtain the semantic representation matrix of the anchor point sample:

[0088] H i ={h i1 ,h i2 ,…,h il ,…,h iL}=f(S i )

[0089] Among them, H i represents the representation matrix of the i-th sentence after being encoded by the pre-trained model (the representation will be continuously updated as the entire model is trained), where h ij represents the vector representation of the jth word; f() represents the pre-training model;

[0090] Use the same pre-training model to obtain the semantic representation matrix of the negative sample

[0091] It should be noted that the pre-trained model used to encode the input sentence can be selected according to actual needs and is not absolutely limited in the embodiments of the present invention. For example, a pre-trained BERT network can be used.

[0092] Next, in order to avoid the problem that traditional enhancement methods will modify the semantics of the original sentence, the semantic representation matrix H i The order is disrupted, some feature columns are masked, and the elements in the entire matrix representation are randomly discarded to obtain the feature-insensitive enhanced representation. The process can be expressed as follows:

[0093] op1=random(shuffling,featurecutoff,dropout),

[0094]

[0095] Among them, op1 represents the first data enhancement method, shuffling represents the order shuffling operation, featurecutoff represents the operation of covering part of the feature column, and dropout represents the random dropping operation; It means the semantically invariant feature-insensitive enhanced representation after data enhancement.

[0096] In step S2, the pre-trained model is used to obtain data-related feature-sensitive enhanced representation based on the weight of each word in the anchor sample in the entire training set and the semantic representation matrix of the anchor sample.

[0097] Considering that the model should be sensitive to data-related features in the input sentence to ensure performance on downstream tasks, the embodiment of the present invention also designs data-related feature-sensitive representation enhancement.

[0098] Specifically, the S2 includes:

[0099] S21, using the TF-IDF method to statistically analyze all sentences in the training set, and obtain the weight α of each word in the anchor sample in the entire training set. ij , and compare it with the preset weight threshold θ. If α ij >θ, perform data enhancement on the word corresponding to the weight;

[0100] S22. Combined with the semantic representation matrix of the anchor sample, data enhancement is performed on the words that require data enhancement as determined in the above step with a certain probability β at the word level using WordNet, and the pre-trained model is used to obtain data-related feature-sensitive enhanced representations. The process can be expressed as follows:

[0101] op2=random(Synonym,Delete,Swap,Noop)

[0102] Among them, op2 represents the second data augmentation method, Synonym represents the synonym replacement operation, Delete represents the random deletion operation, Swap represents the random swap operation, and Noop represents no operation;

[0103]

[0104]

[0105]

[0106] in, Represents the one-hot representation of the lth word in the i-th sentence after data enhancement; Indicates the sentence representation after data enhancement; It represents the data-related feature-sensitive enhanced representation; f() represents the pre-training model, and the pre-training model here can also use the BERT network.

[0107] In step S3, the semantic representation matrices of the anchor samples, negative samples and positive samples, and the semantic representation vectors corresponding to the feature-sensitive enhanced representation are obtained respectively; the semantic representation vectors corresponding to the semantic representation matrices of the anchor samples, negative samples and positive samples are mapped to different metric spaces to obtain a first mapping output result; the semantic representation vectors corresponding to the feature-sensitive enhanced representation are mapped to the final decision space to obtain a second mapping output result.

[0108] This step is essentially feature-insensitive contrast learning for sentences and feature-sensitive learning for data. Specifically, S3 includes:

[0109] S31, using pooling operations to obtain the semantic representation vectors v corresponding to the semantic representation matrices of the anchor sample, negative sample and positive sample respectively i 、

[0110]

[0111] Among them, mean represents average pooling;

[0112] The semantic representation vectors corresponding to the semantic representation matrices of the anchor samples, negative samples and positive samples are mapped to different metric spaces by a first multi-layer perceptron to obtain a first mapping output result z i 、

[0113]

[0114] Among them, MLP1 represents the first multi-layer perceptron;

[0115] S32. For feature-sensitive enhanced representation, considering that the final model needs to judge each word, Bi-LSTM is used to obtain the semantic representation vector corresponding to the feature-sensitive enhanced representation And the initial hidden state of the recurrent neural network uses the semantic representation vector v corresponding to the original anchor point sample i ,

[0116]

[0117] The semantic representation vector corresponding to the feature-sensitive enhanced representation is mapped to the final decision space by a second multi-layer perceptron to obtain a second mapping output result.

[0118]

[0119] Among them, MLP2 represents the second multi-layer perceptron, which is used to map the obtained representation to the final decision space.

[0120] In step S4, a first loss function is constructed based on the first mapping output result; a second loss function is constructed based on the second mapping output result; the first loss function and the second loss function are fused to obtain a final loss function and used to optimize parameters until the model converges.

[0121] In this stage, the model is trained mainly through different optimization objectives.

[0122] Specifically, for feature-insensitive contrastive learning, the embodiment of the present invention adopts the same contrastive learning optimization objective as the existing work, that is, the first loss function is expressed as:

[0123]

[0124] Among them, log represents the logarithmic function; sim represents the similarity function; K represents the number of negative samples, and τ is the temperature coefficient in the first loss function, which is a hyperparameter. It should be noted that the number of negative samples in contrastive learning is generally large, and in the embodiment of the present invention, only the generation method of a single negative sample is introduced, and other negative samples are also generated by the same method.

[0125] For data-related feature-sensitive learning, this project sets an additional prediction task: the model needs to determine which words in the input augmented sentence are task-related words and whether these words are modified. That is, the second loss function is expressed as:

[0126]

[0127] Among them, y′ it and They represent the true label and predicted label of the lth word in the i-th enhanced sentence respectively. The labels are divided into the following four categories: task-related words, modified; task-related words, not modified; task-irrelevant words, modified; task-irrelevant words, not modified.

[0128] The final loss function is expressed as:

[0129] L=μL1+(1-μ)L2

[0130] Here, μ is a hyperparameter.

[0131] After the model converges, the selected anchor point samples are used as the final model output and then applied to specific downstream tasks.

[0132] like Figure 2 As shown, an embodiment of the present invention provides a contrastive learning system based on text sensitivity and semantic invariance, including:

[0133] The first enhancement module is configured to execute S1, determine anchor samples and negative samples in the current training batch, and use a pre-trained model to obtain semantic representation matrices of the anchor samples and negative samples, respectively; perform data enhancement on the semantic representation matrix of the anchor samples to obtain a semantically invariant feature-insensitive enhanced representation, and use it as the semantic representation of the positive sample;

[0134] A second enhancement module is configured to execute S2, and obtain a data-related feature-sensitive enhanced representation using the pre-trained model according to the weight of each word in the anchor sample in the entire training set and the semantic representation matrix of the anchor sample;

[0135] A mapping module is configured to execute S3, obtain the semantic representation matrices of the anchor sample, negative sample, and positive sample, and the semantic representation vectors corresponding to the feature-sensitive enhanced representation, respectively; map the semantic representation vectors corresponding to the semantic representation matrices of the anchor sample, negative sample, and positive sample to different metric spaces to obtain a first mapping output result; and map the semantic representation vector corresponding to the feature-sensitive enhanced representation to a final decision space to obtain a second mapping output result;

[0136] An optimization module is used to execute S4, construct a first loss function based on the first mapping output result; construct a second loss function based on the second mapping output result; and fuse the first loss function and the second loss function to obtain a final loss function and use it to optimize parameters until the model converges.

[0137] An embodiment of the present invention provides a storage medium storing a computer program for contrastive learning based on text sensitivity and semantic invariance, wherein the computer program enables a computer to execute the contrastive learning method described above.

[0138] An embodiment of the present invention further provides an electronic device, including:

[0139] one or more processors;

[0140] Memory; and

[0141] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for executing the contrastive learning method as described above.

[0142] It is understandable that the contrastive learning system based on text sensitivity and semantic invariance, storage medium and electronic device provided in the embodiments of the present invention correspond to the contrastive learning method based on text sensitivity and semantic invariance provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the contrastive learning method and will not be repeated here.

[0143] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0144] Compared with existing methods, the embodiments of the present invention, on the one hand, ensure that the semantics of the input sentence remains unchanged after data enhancement by performing data enhancement at the representation level, thereby more effectively supporting contrastive learning for feature-insensitive learning. On the other hand, by designing task-related or data-related sensitive feature selection methods, the embodiments of the present invention make the model more targeted during feature-sensitive learning, improving the model's adaptability to downstream tasks. Moreover, the sensitive feature selection method is decoupled from the sensitive feature learning process, ensuring that the model can design different sensitive feature selection methods according to specific needs, thereby improving the model's generalization ability and applicability.

[0145] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0146] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A contrastive learning method based on text sensitivity and semantic invariance, characterized in that include: S1. Determine anchor samples and negative samples in the current training batch, and use a pre-training model to obtain semantic representation matrices of the anchor samples and negative samples respectively; Performing data augmentation on the semantic representation matrix of the anchor point sample to obtain a semantically invariant feature-insensitive enhanced representation, and using it as the semantic representation of the positive sample; S2. Based on the weight of each word in the anchor sample in the entire training set and the semantic representation matrix of the anchor sample, the pre-trained model is used to obtain a data-related feature-sensitive enhanced representation; S3. Obtain the semantic representation matrices of the anchor sample, negative sample, and positive sample, and the semantic representation vectors corresponding to the feature-sensitive enhanced representation, respectively; map the semantic representation vectors corresponding to the semantic representation matrices of the anchor sample, negative sample, and positive sample to different metric spaces to obtain a first mapping output result; map the semantic representation vector corresponding to the feature-sensitive enhanced representation to a final decision space to obtain a second mapping output result; S4. Construct a first loss function based on the first mapping output result; construct a second loss function based on the second mapping output result; fuse the first loss function and the second loss function to obtain a final loss function and use it to optimize parameters until the model converges; In S1: the semantic representation matrix H i The order is disrupted, some feature columns are masked, and the elements in the entire matrix representation are randomly discarded to obtain the feature-insensitive enhanced representation. The process can be expressed as follows: op1=random(shuffling,feature cutoff,dropout), Among them, op1 represents the first data augmentation method, shuffling represents the order shuffling operation, feature cutoff represents the operation of covering part of the feature column, and dropout represents the random dropping operation; It means the semantically invariant feature insensitive enhancement representation after data enhancement; The S2 includes: S21, using the TF-IDF method to statistically analyze all sentences in the training set, and obtain the weight α of each word in the anchor sample in the entire training set. ij , and compare it with the preset weight threshold θ. If α ij >θ, perform data enhancement on the word corresponding to the weight; S22. Combined with the semantic representation matrix of the anchor sample, data enhancement is performed on the words that require data enhancement as determined in the above step with a certain probability β at the word level using WordNet, and the pre-trained model is used to obtain data-related feature-sensitive enhanced representations. The process can be expressed as follows: op2=random(Synonym,Delete,Swap,Noop) Among them, op2 represents the second data augmentation method, Synonym represents the synonym replacement operation, Delete represents the random deletion operation, Swap represents the random swap operation, and Noop represents no operation; in, Represents the one-hot representation of the lth word in the i-th sentence after data enhancement; Indicates the sentence representation after data enhancement; represents the data-dependent feature-sensitive enhanced representation; f() represents the pre-trained model.

2. The contrastive learning method based on text sensitivity and semantic invariance according to claim 1, characterized in that In S1: Select other non-anchor samples in the same training batch as negative samples of the anchor samples; Define the input sentence corresponding to the anchor sample as S i ={s i1 , s i2 ,…,s il ,…,s iL }, where s il represents the one-hot representation of the lth word in the i-th sentence, where L represents the length of the sentence; The pre-trained model is used to obtain the semantic representation matrix of the anchor point sample: H i ={h i1 ,h i2 ,...,h il ,...,h iL }=f(S i ) Among them, H i represents the representation matrix of the i-th sentence after encoding by the pre-training model, where h ij represents the vector representation of the jth word; f() represents the pre-training model; Use the same pre-training model to obtain the semantic representation matrix of the negative sample 3. The contrastive learning method based on text sensitivity and semantic invariance according to claim 1, characterized in that The S3 includes: S31, using pooling operations to obtain the semantic representation vectors v corresponding to the semantic representation matrices of the anchor sample, negative sample and positive sample respectively i 、 Among them, mean represents average pooling; The semantic representation vectors corresponding to the semantic representation matrices of the anchor samples, negative samples and positive samples are mapped to different metric spaces by a first multi-layer perceptron to obtain a first mapping output result z i 、 Among them, MLP1 represents the first multi-layer perceptron; S32, using Bi-LSTM to obtain the semantic representation vector corresponding to the feature-sensitive enhanced representation The initial hidden state of the recurrent neural network uses the semantic representation vector v corresponding to the original anchor point sample i , The semantic representation vector corresponding to the feature-sensitive enhanced representation is mapped to the final decision space by a second multi-layer perceptron to obtain a second mapping output result. Among them, MLP2 represents the second multi-layer perceptron.

4. The contrastive learning method based on text sensitivity and semantic invariance according to claim 3, characterized in that In said S4: The first loss function is expressed as: Where log represents the logarithmic function; sim represents the similarity function; K represents the number of negative samples; τ is the temperature coefficient in the first loss function, which is a hyperparameter; The second loss function is expressed as: Among them, y′ it and They represent the true label and predicted label of the lth word in the i-th augmented sentence, respectively. The labels are divided into the following four categories: task-related words, modified; task-related words, not modified; task-irrelevant words, modified; task-irrelevant words, not modified; The final loss function is expressed as: L=μL1+(1-μ)L2 Here, μ is a hyperparameter.

5. The contrastive learning method based on text sensitivity and semantic invariance according to claim 4, characterized in that: The pre-trained model is a pre-trained BERT network.

6. A contrastive learning system based on text sensitivity and semantic invariance, characterized in that The method for performing the contrastive learning method based on text sensitivity and semantic invariance as claimed in claim 1 comprises: The first enhancement module is configured to execute S1, determine anchor samples and negative samples in the current training batch, and use a pre-trained model to obtain semantic representation matrices of the anchor samples and negative samples, respectively; perform data enhancement on the semantic representation matrix of the anchor samples to obtain a semantically invariant feature-insensitive enhanced representation, and use it as the semantic representation of the positive sample; A second enhancement module is configured to execute S2, and obtain a data-related feature-sensitive enhanced representation using the pre-trained model according to the weight of each word in the anchor sample in the entire training set and the semantic representation matrix of the anchor sample; A mapping module is configured to execute S3, obtain the semantic representation matrices of the anchor sample, negative sample, and positive sample, and the semantic representation vectors corresponding to the feature-sensitive enhanced representation, respectively; map the semantic representation vectors corresponding to the semantic representation matrices of the anchor sample, negative sample, and positive sample to different metric spaces to obtain a first mapping output result; and map the semantic representation vector corresponding to the feature-sensitive enhanced representation to a final decision space to obtain a second mapping output result; An optimization module is used to execute S4, construct a first loss function based on the first mapping output result; construct a second loss function based on the second mapping output result; and fuse the first loss function and the second loss function to obtain a final loss function and use it to optimize parameters until the model converges.

7. A storage medium, characterized in that: The computer program for contrastive learning based on text sensitivity and semantic invariance is stored therein, wherein the computer program enables the computer to execute the contrastive learning method based on text sensitivity and semantic invariance according to any one of claims 1 to 5.

8. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the contrastive learning method based on text sensitivity and semantic invariance as described in any one of claims 1 to 5.

Images