A method, device, computer equipment and storage medium for constructing positive and negative sample pairs

Abstract

The embodiments of the present application belong to the field of natural language processing technology, and relate to a method, device, computer equipment and storage medium for constructing positive and negative sample pairs. In addition, the present application also relates to blockchain technology, and the user's target positive and negative sample pairs can be stored in the blockchain. The present application uses the random deactivation function in the hidden layer of the pre-trained model as an augmentation method for constructing sample pairs. Since the augmentation method of the hidden layer is to randomly return some weights or outputs to zero, it effectively solves the problem that traditional augmentation methods such as translation, deletion, insertion, and substitution are prone to introduce negative noise, so that the constructed sample pairs still retain the semantic features of the original text to be augmented, and the distribution of similar samples is very close and the distribution of samples is relatively uniform. At the same time, it provides better sentence vectors for downstream tasks.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a method, device, computer equipment and storage medium for constructing positive and negative sample pairs. Background Art

[0002] In the field of natural language processing, learning a universal sentence vector representation method is a basic but very important task. On the one hand, an excellent sentence vector representation method can represent the semantics of each sentence. On the other hand, sentence vectors that extract important semantic features can better provide training data for downstream tasks such as text classification, text generation, and semantic similarity calculation, thereby improving task performance. In natural language processing, sentence vector conversion models are widely used, and their importance is self-evident.

[0003] The most widely used way to obtain word vectors is to use the pre-trained model CLS or sentence sequence dimension to perform meanpooling to obtain the feature vector of the sentence. These methods will cause the model to be anisotropic, that is, the feature representation of each dimension of the word vector is inconsistent during the model training process, and the obtained sentence vectors cannot be directly compared.

[0004] There is a sentence vector representation method that solves the problem that the feature representations of each dimension of the above word vector are inconsistent and the obtained sentence vectors cannot be directly compared. That is, contrastive learning is used to bring similar sentences closer and push dissimilar sentences apart in the semantic representation space.

[0005] However, the applicant found that traditional sentence vector representation methods using contrastive learning are generally not intelligent, because the common methods of sentence augmentation in the current contrastive learning methods of sentence vector representation are translation, deletion, insertion, substitution, etc., but these are discrete operations, difficult to control and easy to introduce negative noise. Therefore, it is difficult for the model to capture semantic information from these samples through contrastive learning. It can be seen that the traditional sentence vector representation methods using contrastive learning have the problem of being unable to learn the true semantic information of the samples. Summary of the invention

[0006] The purpose of the embodiments of the present application is to propose a method, device, computer equipment and storage medium for constructing positive and negative sample pairs to solve the problem that the traditional sentence vector representation method using contrastive learning cannot learn the true semantic information of the sample.

[0007] In order to solve the above technical problems, the present application embodiment provides a method for constructing positive and negative sample pairs, which adopts the following technical solution:

[0008] When performing model comparison learning training, receiving a sample acquisition request carrying an encoder model to be trained;

[0009] Reading a text database, and obtaining text data to be augmented from the text database;

[0010] Constructing initial positive and negative sample pairs according to the text data to be augmented;

[0011] The initial positive and negative sample pairs are forward propagated according to the random deactivation function of the hidden layer in the encoder model to obtain discrete and uniform target positive and negative sample pairs.

[0012] In order to solve the above technical problems, the embodiment of the present application further provides a positive and negative sample pair construction device, which adopts the following technical solution:

[0013] A request acquisition module, used for receiving a sample acquisition request carrying an encoder model to be trained when performing model comparison learning training;

[0014] A data acquisition module, used for reading a text database and acquiring text data to be augmented from the text database;

[0015] An initial sample construction module, used to construct initial positive and negative sample pairs according to the text data to be augmented;

[0016] The target sample acquisition module is used to perform a forward propagation operation on the initial positive and negative sample pairs according to the random deactivation function of the hidden layer in the encoder model to obtain discrete and uniform target positive and negative sample pairs.

[0017] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0018] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the positive and negative sample pair construction method as described above when executing the computer-readable instructions.

[0019] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0020] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the positive and negative sample pair construction method described above are implemented.

[0021] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0022] The present application provides a method for constructing positive and negative sample pairs, including: receiving a sample acquisition request carrying an encoder model to be trained during model contrast learning training; reading a text database, and acquiring text data to be augmented from the text database; constructing initial positive and negative sample pairs according to the text data to be augmented; and performing forward propagation operations on the initial positive and negative sample pairs according to the random deactivation function of the hidden layer in the encoder model to obtain discrete and uniform target positive and negative sample pairs. The present application utilizes the random deactivation function in the hidden layer of the pre-trained model as an augmentation method for constructing sample pairs. Since the augmentation method of the hidden layer is to randomly return some weights or outputs to zero, it effectively solves the problem that traditional augmentation methods such as translation, deletion, insertion, and substitution are prone to introduce negative noise, so that the constructed sample pairs still retain the semantic features of the original text to be augmented, and the distribution of similar samples is very close and the distribution of samples is relatively uniform. At the same time, it provides better sentence vectors for downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 is a flowchart of the implementation of the positive and negative sample pair construction method provided in Example 1 of the present application;

[0025] Figure 2 It is a flowchart of a specific implementation method of the method for constructing positive and negative sample pairs for establishing a prediction model provided in Example 1 of the present application;

[0026] Figure 3 It is a schematic diagram of the process of constructing positive and negative sample pairs for supervised tasks provided in Example 1 of the present application;

[0027] Figure 4 It is a flowchart of a specific implementation method of the method for constructing positive and negative sample pairs for establishing a description model provided in Example 1 of the present application;

[0028] Figure 5 It is a schematic diagram of the process of constructing positive and negative sample pairs for unsupervised tasks provided in Example 1 of the present application;

[0029] Figure 6 This is a schematic diagram of the structure of the positive and negative sample pair construction device provided in Example 2 of the present application:

[0030] Figure 7It is a structural schematic diagram of a specific implementation of a positive and negative sample pair construction device for establishing a prediction model provided in Example 2 of the present application;

[0031] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0035] like Figure 1 As shown, a flowchart of the implementation of the positive and negative sample pair construction method provided in Example 1 of the present application is shown. For the sake of convenience of explanation, only the parts related to the present application are shown.

[0036] The above-mentioned method for constructing positive and negative sample pairs includes the following steps:

[0037] Step S101: When performing model comparison learning training, a sample acquisition request carrying an encoder model to be trained is received.

[0038] In the present application embodiment, contrastive learning is a method for describing similar and different tasks for ML models. Using this method, machine learning models can be trained to distinguish similar and different images. The internal workings of contrastive learning can be expressed as a score function, which is a measure of the similarity between two features.

[0039] Step S102: reading a text database, and obtaining text data to be augmented from the text database.

[0040] In an embodiment of the present application, the text database is mainly used to store the above-mentioned text data to be augmented, wherein the text data to be augmented may be unlabeled text data suitable for unsupervised learning, and the text data to be augmented may also be labeled text data suitable for supervised learning. In practical applications, the corresponding text data to be augmented may be selected according to the learning objectives of the above-mentioned encoder model to be trained.

[0041] Step S103: constructing initial positive and negative sample pairs according to the text data to be augmented.

[0042] In the embodiment of the present application, constructing the initial positive and negative sample pairs means taking the text data with an associated relationship as the same positive sample group, taking the text data with no associated relationship with the above positive sample group as the negative sample, and integrating the above positive sample group and the above negative sample to obtain the above initial positive and negative sample pairs. Among them, when the text data to be augmented is unlabeled text data, since there is no associated relationship between the text data, the same text data is taken as the unlabeled positive sample in the unlabeled positive sample group, and the text data different from the unlabeled positive sample is taken as the unlabeled negative sample.

[0043] Step S104 performs a forward propagation operation on the initial positive and negative sample pairs according to the random deactivation function of the hidden layer in the encoder model to obtain discrete and uniform target positive and negative sample pairs.

[0044] In the embodiments of the present application, random dropout is a method for optimizing an artificial neural network with a deep structure. During the learning process, some weights or outputs of the hidden layer are randomly reset to zero to reduce the co-dependence between nodes, thereby achieving regularization of the neural network and reducing its structural risk.

[0045] In the embodiment of the present application, forward propagation refers to performing weighted sum operations on the nodes of a layer and the corresponding connection weights, adding a bias term to the result, and then passing through a nonlinear function (i.e., activation function), such as ReLu, sigmoid, etc., and the result obtained is the output of the nodes of the next layer. Starting from the first layer (input layer), this method is continuously used layer by layer to finally obtain the output layer result.

[0046] In an embodiment of the present application, a method for constructing positive and negative sample pairs is provided, including: when performing model contrast learning training, receiving a sample acquisition request carrying an encoder model to be trained; reading a text database, and obtaining text data to be augmented in the text database; constructing initial positive and negative sample pairs according to the text data to be augmented; performing forward propagation operations on the initial positive and negative sample pairs according to the random deactivation function of the hidden layer in the encoder model, and obtaining discrete and uniform target positive and negative sample pairs. The present application utilizes the random deactivation function in the hidden layer of the pre-trained model as an augmentation method for constructing sample pairs. Since the augmentation method of the hidden layer is to randomly return some weights or outputs to zero, it effectively solves the problem that traditional augmentation methods such as translation, deletion, insertion, and substitution are prone to introduce negative noise, so that the constructed sample pairs still retain the semantic features of the original text to be augmented, and the distribution of similar samples is very close and the distribution of samples is relatively uniform. At the same time, it provides better sentence vectors for downstream tasks.

[0047] Continue reading Figure 2 , shows a flowchart of a specific implementation of the method for constructing positive and negative sample pairs for establishing a prediction model provided in Example 1 of the present application. For the sake of ease of explanation, only the part related to the present application is shown.

[0048] In some optional implementations of this embodiment, the above step S102 specifically includes: step S201; step S103 specifically includes: step S202, step S203, step S204 and step S205.

[0049] Step S201: If the learning goal of the encoder model to be trained is to establish a prediction model, annotated text data is obtained from a text database, and the annotated text data is used as the text data to be augmented.

[0050] In the embodiments of the present application, machine learning models are basically divided into two categories: supervised learning and unsupervised learning. Supervised learning is used to establish a prediction model. Since the prediction model has clear guidance on "what to learn" and "how to learn", the process of training a prediction model is called supervised learning. Unsupervised learning is used to establish a descriptive model, that is, there is no learning goal, and it is a process of training a descriptive model.

[0051] In an embodiment of the present application, whether to establish a prediction model can be confirmed based on the learning type of the above-mentioned encoder model to be trained. Specifically, it can be distinguished based on the characteristics of the output layer of the above-mentioned encoder model to be trained. When the output layer is to determine a clear target such as a numerical value or a classification result, it can be confirmed that a prediction model is to be established. When the above-mentioned output layer is clustering, pattern recognition, etc., it can be confirmed that a description model is to be established.

[0052] In an embodiment of the present application, when the learning goal is to establish a prediction model, text data carrying annotation information is required as training data for the encoder model to be trained. Therefore, when obtaining the text data to be augmented, it is necessary to obtain annotated text data.

[0053] In an embodiment of the present application, annotated text data refers to text data annotated with the association relationship between sentences, and the association relationship is mainly used to provide learning direction for the encoder model to be trained, wherein the association relationship can be an implication relationship or a conflict relationship.

[0054] Step S202: extracting a first sentence to be learned from the annotated text data, and taking the first sentence to be learned as a first predicted positive sample.

[0055] In the embodiment of the present application, the first sentence to be learned refers to any sentence in the annotated text data, which serves as the original original data in the positive sample suitable for training the prediction model, that is, the above-mentioned first prediction positive sample.

[0056] Step S203: extracting sentences that have an implication relationship with the first sentence to be learned from the annotated text data as second predicted positive samples.

[0057] Step S204: extracting sentences that conflict with the first sentence to be learned from the annotated text data as predicted negative samples.

[0058] Step S205: combining the first predicted positive sample, the second predicted positive sample and the predicted negative sample to obtain an initial positive and negative sample pair.

[0059] In the examples of this application, refer to Figure 3 , which shows a schematic diagram of the construction process of positive and negative sample pairs for supervised tasks provided in the first embodiment of the present application, where each sentence is represented as s i , taking sentences with implicit relationships as positive samples Sentences and other sentences that conflict with it are used as negative samples Depend on Constitute the above initial positive and negative sample pairs.

[0060] In some optional implementations of the embodiments of the present application, there is a natural language inference dataset whose annotated text data is composed of the Stanford Natural Language Inference corpus (SNLL) and the Multi-Genre Natural Language Inference corpus (MNLI).

[0061] In some optional implementations of the embodiments of the present application, after the above step S205, the following steps are also included:

[0062] The prediction model training is performed on the encoder model to be trained according to the prediction loss function, and the prediction loss function is expressed as:

[0063]

[0064] Wherein, E represents the first predicted positive sample; E + represents the second predicted positive sample; E - represents the predicted negative sample; i and j represent the sample sequence numbers in the sample set respectively; N represents the total number of sample pairs in the sample set; and r is a hyperparameter that plays a role in scaling.

[0065] In practical applications, if the sample pair of the sample set is Then the total number of sample pairs in the sample set is n. When the encoder model to be trained is trained When this sample pair is selected, any set of sample pairs will be selected from the remaining sample pairs in the sample set. (n≠1) is used as the prediction loss function above. and To train the prediction model.

[0066] Continue reading Figure 4 , shows a flowchart of a specific implementation of the method for constructing positive and negative sample pairs for establishing a description model provided in Example 1 of the present application. For the sake of ease of explanation, only the parts related to the present application are shown.

[0067] In some optional implementations of this embodiment, the above step S102 specifically includes: step S401; step S103 specifically includes: step S402, step S403 and step S404.

[0068] Step S401: if the learning goal of the encoder model to be trained is to establish a description model, unlabeled text data is obtained from a text database, and the unlabeled text data is used as the text data to be augmented.

[0069] Step S402: extracting a second sentence to be learned from the unlabeled text data, and using the second sentence to be learned as a first description positive sample and a second description positive sample.

[0070] In the embodiment of the present application, the second sentence to be learned refers to any sentence in the unlabeled text data, which serves as the original data in the positive sample suitable for training the description model, that is, the above-mentioned first description positive sample.

[0071] In the embodiment of the present application, since it is impossible to confirm whether the sentences in the unlabeled text data are similar, when constructing the positive sample, the same sentence is first used as two positive samples suitable for training the description model. Since the semantic features between the two positive samples are completely consistent, they can be augmented through subsequent forward propagation operations to form positive samples with different sentences and certain implicit relationships.

[0072] Step S403: extracting sentences inconsistent with the second sentence to be learned from the standard text data as description negative samples.

[0073] Step S404: combining the first description positive sample, the second description positive sample and the description negative sample to obtain an initial positive and negative sample pair.

[0074] In the examples of this application, refer to Figure 5 , which shows a schematic diagram of the process of constructing positive and negative sample pairs for unsupervised tasks provided in the first embodiment of the present application, using two sentence vectors obtained from the same sentence as a positive sample pair In addition, different sentences are selected to obtain sentence vectors through the encoder model As negative samples. As the initial positive and negative sample pairs in contrastive learning.

[0075] In some optional implementations of the embodiments of the present application, after the above step S404, the following steps are also included:

[0076] The encoder model to be trained is trained according to the description loss function, and the description loss function is expressed as:

[0077]

[0078] Among them, E m represents the first description positive sample; E m′ represents the second description positive sample; E m″ represents the described negative sample; i and j represent the sample sequence numbers in the sample set respectively; N represents the total number of sample pairs in the sample set; and r is a hyperparameter that plays a role in scaling.

[0079] In practical applications, if the sample pair of the sample set is Then the total number of sample pairs in the sample set is n. When the encoder model to be trained is trained When this sample pair is selected, any set of sample pairs will be selected from the remaining sample pairs in the sample set. (n≠1) As described above, the loss function To train the prediction model.

[0080] In summary, the present application provides a method for constructing positive and negative sample pairs, including: receiving a sample acquisition request carrying an encoder model to be trained during model contrast learning training; reading a text database, and acquiring text data to be augmented in the text database; constructing initial positive and negative sample pairs according to the text data to be augmented; performing forward propagation operations on the initial positive and negative sample pairs according to the random deactivation function of the hidden layer in the encoder model to obtain discrete and uniform target positive and negative sample pairs. The present application utilizes the random deactivation function in the hidden layer of the pre-trained model as an augmentation method for constructing sample pairs. Since the augmentation method of the hidden layer is to randomly return some weights or outputs to zero, it effectively solves the problem that traditional augmentation methods such as translation, deletion, insertion, and substitution are prone to introduce negative noise, so that the constructed sample pairs still retain the semantic features of the original text to be augmented, and the distribution of similar samples is very close and the distribution of samples is relatively uniform. At the same time, it provides better sentence vectors for downstream tasks. At the same time, the present application can also confirm the type of model learning according to the learning objectives of the encoder model to be trained, and construct matching samples based on different types, so that it is suitable for unsupervised or supervised tasks in natural language processing, greatly improving the compatibility of the present application.

[0081] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target positive and negative sample pairs, the above-mentioned target positive and negative sample pairs can also be stored in a blockchain node.

[0082] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0083] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0084] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0085] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0086] Embodiment 2

[0087] Further references Figure 6 , as a response to the above Figure 1 The present application provides an embodiment of a positive and negative sample pair construction device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0088] like Figure 6 As shown, the positive and negative sample pair construction device 100 of this embodiment includes: a request acquisition module 110 , a data acquisition module 120 , an initial sample construction module 130 and a target sample acquisition module 140 .

[0089] in:

[0090] A request acquisition module 110, used to receive a sample acquisition request carrying an encoder model to be trained when performing model comparison learning training;

[0091] A data acquisition module 120 is used to read a text database and acquire text data to be augmented from the text database;

[0092] An initial sample construction module 130, used to construct initial positive and negative sample pairs according to the text data to be augmented;

[0093] The target sample acquisition module 140 is used to perform a forward propagation operation on the initial positive and negative sample pairs according to the random deactivation function of the hidden layer in the encoder model to obtain discrete and uniform target positive and negative sample pairs.

[0094] In the present application embodiment, contrastive learning is a method for describing similar and different tasks for ML models. Using this method, machine learning models can be trained to distinguish similar and different images. The internal workings of contrastive learning can be expressed as a score function, which is a measure of the similarity between two features.

[0095] In an embodiment of the present application, the text database is mainly used to store the above-mentioned text data to be augmented, wherein the text data to be augmented may be unlabeled text data suitable for unsupervised learning, and the text data to be augmented may also be labeled text data suitable for supervised learning. In practical applications, the corresponding text data to be augmented may be selected according to the learning objectives of the above-mentioned encoder model to be trained.

[0096] In the embodiment of the present application, constructing the initial positive and negative sample pairs means taking the text data with an associated relationship as the same positive sample group, taking the text data with no associated relationship with the above positive sample group as the negative sample, and integrating the above positive sample group and the above negative sample to obtain the above initial positive and negative sample pairs. Among them, when the text data to be augmented is unlabeled text data, since there is no associated relationship between the text data, the same text data is taken as the unlabeled positive sample in the unlabeled positive sample group, and the text data different from the unlabeled positive sample is taken as the unlabeled negative sample.

[0097] In the embodiments of the present application, random dropout is a method for optimizing an artificial neural network with a deep structure. During the learning process, some weights or outputs of the hidden layer are randomly reset to zero to reduce the co-dependence between nodes, thereby achieving regularization of the neural network and reducing its structural risk.

[0098] In the embodiment of the present application, forward propagation refers to performing weighted sum operations on the nodes of a layer and the corresponding connection weights, adding a bias term to the result, and then passing through a nonlinear function (i.e., activation function), such as ReLu, sigmoid, etc., and the result obtained is the output of the nodes of the next layer. Starting from the first layer (input layer), this method is continuously used layer by layer to finally obtain the output layer result.

[0099] In an embodiment of the present application, a method for constructing positive and negative sample pairs is provided, including: when performing model contrast learning training, receiving a sample acquisition request carrying an encoder model to be trained; reading a text database, and acquiring text data to be augmented in the text database; constructing initial positive and negative sample pairs according to the text data to be augmented; performing a forward propagation operation on the initial positive and negative sample pairs according to the random inactivation function of the hidden layer in the encoder model to obtain discrete and uniform target positive and negative sample pairs.

[0100] In an embodiment of the present application, a positive and negative sample pair construction device is provided, including: a request acquisition module 110, which is used to receive a sample acquisition request carrying an encoder model to be trained when performing model contrast learning training; a data acquisition module 120, which is used to read a text database and obtain text data to be augmented in the text database; an initial sample construction module 130, which is used to construct an initial positive and negative sample pair according to the text data to be augmented; a target sample acquisition module 140, which is used to perform a forward propagation operation on the initial positive and negative sample pair according to the random deactivation function of the hidden layer in the encoder model to obtain a discrete and uniform target positive and negative sample pair. The present application uses the random deactivation function in the hidden layer of the pre-trained model as an augmentation method for constructing sample pairs. Since the augmentation method of the hidden layer is to randomly return some weights or outputs to zero, it effectively solves the problem that traditional augmentation methods such as translation, deletion, insertion, and substitution are prone to introduce negative noise, so that the constructed sample pairs still retain the semantic features of the original text to be augmented, and their similar sample distributions are very close and the sample distribution is relatively uniform. At the same time, it provides a better sentence vector for downstream tasks.

[0101] Continue reading Figure 7, shows a structural schematic diagram of a specific implementation of a positive and negative sample pair construction device for establishing a prediction model provided in Example 2 of the present application. For the sake of ease of explanation, only the parts related to the present application are shown.

[0102] In some optional implementations of this embodiment, the data acquisition module 120 includes: a first data acquisition submodule 121; the initial sample construction module 130 includes: a first positive sample acquisition submodule 131, a second positive sample acquisition submodule 132, a predicted negative sample acquisition submodule 133 and a first initial sample acquisition submodule 134. Among them:

[0103] A first data acquisition submodule 121 is used for acquiring annotated text data from a text database and using the annotated text data as text data to be augmented if the learning goal of the encoder model to be trained is to establish a prediction model;

[0104] A first positive sample acquisition submodule 131 is used to extract a first sentence to be learned from the annotated text data, and use the first sentence to be learned as a first predicted positive sample;

[0105] The second positive sample acquisition submodule 132 is used to extract sentences that have an implication relationship with the first sentence to be learned from the annotated text data as second predicted positive samples;

[0106] A prediction negative sample acquisition submodule 133 is used to extract sentences that conflict with the first sentence to be learned from the annotated text data as prediction negative samples;

[0107] The first initial sample acquisition submodule 134 is used to combine the first predicted positive sample, the second predicted positive sample and the predicted negative sample to obtain an initial positive and negative sample pair.

[0108] In the embodiments of the present application, machine learning models are basically divided into two categories: supervised learning and unsupervised learning. Supervised learning is used to establish a prediction model. Since the prediction model has clear guidance on "what to learn" and "how to learn", the process of training a prediction model is called supervised learning. Unsupervised learning is used to establish a descriptive model, that is, there is no learning goal, and it is a process of training a descriptive model.

[0109] In an embodiment of the present application, whether to establish a prediction model can be confirmed based on the learning type of the above-mentioned encoder model to be trained. Specifically, it can be distinguished based on the characteristics of the output layer of the above-mentioned encoder model to be trained. When the output layer is to determine a clear target such as a numerical value or a classification result, it can be confirmed that a prediction model is to be established. When the above-mentioned output layer is clustering, pattern recognition, etc., it can be confirmed that a description model is to be established.

[0110] In an embodiment of the present application, when the learning goal is to establish a prediction model, text data carrying annotation information is required as training data for the encoder model to be trained. Therefore, when obtaining the text data to be augmented, it is necessary to obtain annotated text data.

[0111] In an embodiment of the present application, annotated text data refers to text data annotated with the association relationship between sentences, and the association relationship is mainly used to provide learning direction for the encoder model to be trained, wherein the association relationship can be an implication relationship or a conflict relationship.

[0112] In the embodiment of the present application, the first sentence to be learned refers to any sentence in the annotated text data, which serves as the original original data in the positive sample suitable for training the prediction model, that is, the above-mentioned first prediction positive sample.

[0113] In the examples of this application, refer to Figure 3 , which shows a schematic diagram of the construction process of positive and negative sample pairs for supervised tasks provided in the first embodiment of the present application, where each sentence is represented as s i , taking sentences with implicit relationships as positive samples Sentences and other sentences that conflict with it are used as negative samples Depend on Constitute the above initial positive and negative sample pairs.

[0114] In some optional implementations of the embodiments of the present application, there is a natural language inference dataset whose annotated text data is composed of the Stanford Natural Language Inference corpus (SNLL) and the Multi-Genre Natural Language Inference corpus (MNLI).

[0115] In some optional implementations of the embodiments of the present application, prediction model training is performed on the encoder model to be trained according to the prediction loss function, and the prediction loss function is expressed as:

[0116]

[0117] Wherein, E represents the first predicted positive sample; E + represents the second predicted positive sample; E - represents the predicted negative sample; i and j represent the sample sequence numbers in the sample set respectively; N represents the total number of sample pairs in the sample set; and r is a hyperparameter that plays a role in scaling.

[0118] In practical applications, if the sample pair of the sample set is Then the total number of sample pairs in the sample set is n. When the encoder model to be trained is trained When this sample pair is selected, any set of sample pairs will be selected from the remaining sample pairs in the sample set. (n≠1) is used as the prediction loss function above. and To train the prediction model.

[0119] In some optional implementations of the embodiments of the present application, the data acquisition module 120 includes: a second data acquisition submodule; the initial sample construction module 130 includes: a third positive sample acquisition submodule, a description negative sample acquisition submodule and a second initial sample acquisition submodule. Among them:

[0120] A second data acquisition submodule is used for acquiring unlabeled text data from a text database if the learning goal of the encoder model to be trained is to establish a description model, and using the unlabeled text data as the text data to be augmented;

[0121] A third positive sample acquisition submodule is used to extract a second sentence to be learned from the unlabeled text data, and use the second sentence to be learned as a first description positive sample and a second description positive sample;

[0122] A description negative sample acquisition submodule is used to extract sentences that are inconsistent with the second sentence to be learned from the standard text data as description negative samples;

[0123] The second initial sample acquisition submodule is used to combine the first description positive sample, the second description positive sample and the description negative sample to obtain an initial positive and negative sample pair.

[0124] In the embodiment of the present application, the second sentence to be learned refers to any sentence in the unlabeled text data, which serves as the original data in the positive sample suitable for training the description model, that is, the above-mentioned first description positive sample.

[0125] In the embodiment of the present application, since it is impossible to confirm whether the sentences in the unlabeled text data are similar, when constructing the positive sample, the same sentence is first used as two positive samples suitable for training the description model. Since the semantic features between the two positive samples are completely consistent, they can be augmented through subsequent forward propagation operations to form positive samples with different sentences and certain implicit relationships.

[0126] In the examples of this application, refer to Figure 5 , which shows a schematic diagram of the process of constructing positive and negative sample pairs for unsupervised tasks provided in the first embodiment of the present application, using two sentence vectors obtained from the same sentence as a positive sample pair In addition, different sentences are selected to obtain sentence vectors through the encoder model As negative samples. As the initial positive and negative sample pairs in contrastive learning.

[0127] In some optional implementations of the embodiments of the present application, the encoder model to be trained is subjected to description model training according to the description loss function, and the description loss function is expressed as:

[0128]

[0129] Among them, E m represents the first description positive sample; E m′ represents the second description positive sample; E m″ represents the described negative sample; i and j represent the sample sequence numbers in the sample set respectively; N represents the total number of sample pairs in the sample set; and r is a hyperparameter that plays a role in scaling.

[0130] In practical applications, if the sample pair of the sample set is Then the total number of sample pairs in the sample set is n. When the encoder model to be trained is trained When this sample pair is selected, any set of sample pairs will be selected from the remaining sample pairs in the sample set. (n≠1) As described above, the loss function To train the prediction model.

[0131] In summary, the present application provides a positive and negative sample pair construction device, including: a request acquisition module 110, which is used to receive a sample acquisition request carrying an encoder model to be trained when performing model comparison learning training; a data acquisition module 120, which is used to read a text database and obtain text data to be augmented in the text database; an initial sample construction module 130, which is used to construct an initial positive and negative sample pair according to the text data to be augmented; a target sample acquisition module 140, which is used to perform a forward propagation operation on the initial positive and negative sample pairs according to the random deactivation function of the hidden layer in the encoder model to obtain a discrete and uniform target positive and negative sample pair. The present application uses the random deactivation function in the hidden layer of the pre-trained model as an augmentation method for constructing sample pairs. Since the augmentation method of the hidden layer is to randomly return some weights or outputs to zero, it effectively solves the problem that traditional augmentation methods such as translation, deletion, insertion, and substitution are prone to introduce negative noise, so that the constructed sample pairs still retain the semantic features of the original text to be augmented, and their similar sample distributions are very close and the sample distribution is relatively uniform. At the same time, it provides a better sentence vector for downstream tasks. At the same time, the present application can also confirm the type of model learning according to the learning objectives of the encoder model to be trained, and construct matching samples based on different types, so that it is suitable for unsupervised or supervised tasks in natural language processing, greatly improving the compatibility of the present application.

[0132] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0133] The computer device 200 includes a memory 210, a processor 220, and a network interface 230 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 200 having components 210-230, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0134] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0135] The memory 210 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 210 can be an internal storage unit of the computer device 200, such as a hard disk or memory of the computer device 200. In other embodiments, the memory 210 can also be an external storage device of the computer device 200, such as a plug-in hard disk equipped on the computer device 200, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 210 can also include both the internal storage unit of the computer device 200 and its external storage device. In this embodiment, the memory 210 is generally used to store an operating system and various application software installed on the computer device 200, such as computer-readable instructions for a method for constructing positive and negative sample pairs, etc. In addition, the memory 210 can also be used to temporarily store various data that have been output or are to be output.

[0136] The processor 220 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 220 is generally used to control the overall operation of the computer device 200. In this embodiment, the processor 220 is used to run computer-readable instructions stored in the memory 210 or process data, such as running computer-readable instructions of the positive and negative sample pair construction method.

[0137] The network interface 230 may include a wireless network interface or a wired network interface. The network interface 230 is generally used to establish a communication connection between the computer device 200 and other electronic devices.

[0138] The computer device provided by the present application utilizes the random deactivation function in the hidden layer of the pre-trained model as an augmentation method for constructing sample pairs. Since the augmentation method of the hidden layer is to randomly return some weights or outputs to zero, the problem that traditional augmentation methods such as translation, deletion, insertion, and substitution easily introduce negative noise is effectively solved, so that the constructed sample pairs still retain the semantic features of the original text to be augmented, and the distribution of similar samples is very close and the distribution of samples is relatively uniform. At the same time, better sentence vectors are provided for downstream tasks.

[0139] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the positive and negative sample pair construction method as described above.

[0140] The computer-readable storage medium provided by the present application utilizes the random deactivation function in the hidden layer of the pre-trained model as a method for constructing an augmentation method for sample pairs. Since the augmentation method of the hidden layer is to randomly return some weights or outputs to zero, the problem that traditional augmentation methods such as translation, deletion, insertion, and substitution easily introduce negative noise is effectively solved, so that the constructed sample pairs still retain the semantic features of the original text to be augmented, and the distribution of similar samples is very close and the distribution of samples is relatively uniform. At the same time, better sentence vectors are provided for downstream tasks.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0142] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for constructing positive and negative sample pairs, characterized in that: The method comprises the following steps: When performing model comparison learning training, receiving a sample acquisition request carrying an encoder model to be trained; Reading a text database, and obtaining text data to be augmented from the text database; Constructing initial positive and negative sample pairs according to the text data to be augmented; Performing a forward propagation operation on the initial positive and negative sample pairs according to the random deactivation function of the hidden layer in the encoder model to obtain discrete and uniform target positive and negative sample pairs; The step of reading a text database and obtaining the text data to be augmented in the text database specifically includes the following steps: If the learning goal of the encoder model to be trained is to establish a description model, unlabeled text data is obtained from the text database, and the unlabeled text data is used as the text data to be augmented; The step of constructing initial positive and negative sample pairs according to the text data to be augmented specifically includes the following steps: Extracting a second sentence to be learned from the unlabeled text data, and using the second sentence to be learned as a first description positive sample and a second description positive sample; Extracting sentences inconsistent with the second sentence to be learned from the unlabeled text data as description negative samples; The initial positive and negative sample pairs are obtained by combining the first description positive sample, the second description positive sample and the description negative sample.

2. The method for constructing positive and negative sample pairs according to claim 1, characterized in that: The step of reading a text database and obtaining the text data to be augmented in the text database specifically includes the following steps: If the learning goal of the encoder model to be trained is to establish a prediction model, then annotated text data is obtained from the text database, and the annotated text data is used as the text data to be augmented; The step of constructing initial positive and negative sample pairs according to the text data to be augmented specifically includes the following steps: Extracting a first sentence to be learned from the annotated text data, and using the first sentence to be learned as a first predicted positive sample; Extracting sentences that have an implication relationship with the first sentence to be learned from the annotated text data as second predicted positive samples; Extracting sentences that conflict with the first sentence to be learned from the annotated text data as predicted negative samples; The first predicted positive sample, the second predicted positive sample, and the predicted negative sample are combined to obtain the initial positive and negative sample pairs.

3. The method for constructing positive and negative sample pairs according to claim 2, characterized in that: The annotated text data is a natural language inference dataset consisting of Stanford natural language inference corpus and multi-genre natural language inference corpus.

4. The method for constructing positive and negative sample pairs according to claim 2, characterized in that: After the step of combining the first predicted positive sample, the second predicted positive sample and the predicted negative sample to obtain the initial positive and negative sample pairs, the following steps are also included: The encoder model to be trained is trained according to the prediction loss function, and the prediction loss function is expressed as: Wherein, E represents the first predicted positive sample; E + represents the second predicted positive sample; E - represents the predicted negative sample; i and j represent the sample sequence numbers in the sample set respectively; N represents the total number of sample pairs in the sample set; and r is a hyperparameter that plays a role in scaling.

5. The method for constructing positive and negative sample pairs according to claim 1, characterized in that: After the step of combining the first description positive sample, the second description positive sample and the description negative sample to obtain the initial positive and negative sample pair, the following step is also included: The encoder model to be trained is trained according to the description loss function, and the description loss function is expressed as: Among them, E m represents the first description positive sample; E m ′ represents the second description positive sample; E m ″ represents the described negative sample; i and j represent the sample serial numbers in the sample set respectively; N represents the total number of sample pairs in the sample set; and r is a hyperparameter.

6. A positive and negative sample pair construction device, characterized in that: The device comprises: A request acquisition module, used for receiving a sample acquisition request carrying an encoder model to be trained when performing model comparison learning training; A data acquisition module, used for reading a text database and acquiring text data to be augmented from the text database; An initial sample construction module, used to construct initial positive and negative sample pairs according to the text data to be augmented; A target sample acquisition module, used for performing a forward propagation operation on the initial positive and negative sample pairs according to the random deactivation function of the hidden layer in the encoder model to obtain discrete and uniform target positive and negative sample pairs; The data acquisition module includes: a second data acquisition submodule, and the initial sample construction module includes: a third positive sample acquisition submodule, a description negative sample acquisition submodule and a second initial sample acquisition submodule, wherein: The second data acquisition submodule is used for acquiring unlabeled text data from the text database and using the unlabeled text data as the text data to be augmented if the learning goal of the encoder model to be trained is to establish a description model; The third positive sample acquisition submodule is used to extract the second sentence to be learned from the unlabeled text data, and use the second sentence to be learned as the first description positive sample and the second description positive sample; The description negative sample acquisition submodule is used to extract sentences inconsistent with the second sentence to be learned from the unlabeled text data as description negative samples; The second initial sample acquisition submodule is used to combine the first description positive sample, the second description positive sample and the description negative sample to obtain the initial positive and negative sample pairs.

7. The positive and negative sample pair construction device according to claim 6, characterized in that: The data acquisition module includes: a first data acquisition submodule; the initial sample construction module includes: a first positive sample acquisition submodule, a second positive sample acquisition submodule, a predicted negative sample acquisition submodule and a first initial sample acquisition submodule, wherein: A first data acquisition submodule, configured to acquire annotated text data from the text database and use the annotated text data as the text data to be augmented if the learning goal of the encoder model to be trained is to establish a prediction model; A first positive sample acquisition submodule, used for extracting a first sentence to be learned from the annotated text data, and using the first sentence to be learned as a first predicted positive sample; A second positive sample acquisition submodule, used to extract, from the annotated text data, sentences that have an implication relationship with the first sentence to be learned as second predicted positive samples; A prediction negative sample acquisition submodule, used to extract, from the annotated text data, sentences that conflict with the first sentence to be learned as prediction negative samples; The first initial sample acquisition submodule is used to combine the first predicted positive sample, the second predicted positive sample and the predicted negative sample to obtain the initial positive and negative sample pairs.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the positive-negative sample pair construction method according to any one of claims 1 to 5 when executing the computer-readable instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for constructing positive and negative sample pairs according to any one of claims 1 to 5 are implemented.

Images