Unsupervised Document Representation Learning Using Contrastive Expansion
The contrastive learning approach with data augmentation techniques addresses data scarcity in deep learning for natural language processing by generating semantically preserved document representations, enhancing accuracy and reducing errors in classification tasks.
Patent Information
- Application Number
- JP2023529085
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-11-17
- Filing Date
- 2021-11-18
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2041-11-18
AI Technical Summary
Existing deep learning methods for natural language processing face challenges due to data scarcity, which affects the quality of document representations and hinders performance in tasks like classification and clustering.
A contrastive learning approach using data augmentation techniques to generate additional documents through methods such as synonym substitution and back-translation, enhancing the training data set while preserving semantic meaning, and incorporating a contrastive loss function to optimize document embeddings.
Improves classification accuracy by up to 7.6% and reduces classification error rates by up to 6.4%, achieving performance comparable to fully supervised methods in unsupervised document representation learning.
Smart Images

Figure 0007578821000056 
Figure 0007578821000057 
Figure 0007578821000058
Abstract
Description
[Technical field]
[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 116,215, filed November 29, 2020, and U.S. Patent Application No. 17 / 528,394, filed November 27, 2021, the disclosures of which are incorporated herein in their entireties.
[0002] The present invention relates to training neural networks, and more particularly to unsupervised training using a contrastive learning approach with data augmentation techniques. [Background technology]
[0003] Deep learning is a branch of machine learning in which computers use deep neural networks to gradually learn to represent and recognize things. When a neural network has multiple hidden layers, it is sometimes called deep.
[0004] Word embeddings are mappings of words to a numerical vector space. Word vectors generated by algorithms such as word2vec map high-dimensional original word representations to a lower-dimensional original vector space. Word embeddings are used in natural language processing (NLP) tasks where machine learning models rely on vector representations as input. The representations may provide semantic and syntactic information about words that can improve the performance of neural networks.
[0005] The bag-of-words approach represents text as a set of words (vocabulary) with no grammatical or order information. The bag-of-words approach can be a one-dimensional vector with length equal to the number of words in the set, where a non-zero value at a position in the vector indicates the presence of that word in the set. The value at a position in the vector may also indicate the number of times the word occurs. In order to preserve some word order information, a bag-of-n-grams approach may be used, where short sequences of words can be represented by vectors, not just individual words.
[0006] Word sense disambiguation (WSD) is the problem of determining which "senses" (meanings) of a word are activated by its use in a particular context. Given a word and its possible senses, as defined in a dictionary, a system can classify occurrences of the word in context into one or more of its semantic classes. In information extraction and text mining, WSD can be included in the accurate analysis of text in many applications. Summary of the Invention
[0007] According to one aspect of the present invention, a method for augmenting a dataset is provided, the method including: providing an original document to a data augmentation generator to generate one or more augmented documents, computing a contrastive loss between the original document and the one or more augmented documents, and training a neural network with the original document and the one or more augmented documents.
[0008] According to another aspect of the present invention, there is provided a system for augmenting a dataset, the system including one or more processors, a memory operatively connected to the one or more processors, a data augmentation generator stored in the memory and configured to generate one or more augmented documents from an original document, and a loss calculator configured to calculate a contrastive loss between the original document and the one or more augmented documents.
[0009] According to another aspect of the invention, there is provided a computer program product for augmenting a dataset, the computer program product being computer readable and causing a computer to receive an original document into a data augmentation generator to generate one or more augmented documents, calculate a contrastive loss between the original document and the one or more augmented documents, and train a neural network using the original document and the one or more augmented documents.
[0010] These and other features and advantages will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings.
[0011] In the present disclosure, preferred embodiments are described in detail with reference to the following drawings, as described below. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a block / flow diagram illustrating a high-level system / method for utilizing augmented documents for representation learning according to one embodiment of the present invention.
[0013] [Diagram 2] FIG. 2 is a block / flow diagram illustrating a Document Embedding with Contrastive Augmentation (DECA) system / method according to one embodiment of the present invention.
[0014] [Diagram 3]FIG. 3 is a block / flow diagram illustrating a neural network for utilizing augmented documents for representation learning, according to one embodiment of the present invention.
[0015] [Figure 4] FIG. 4 is a block / flow diagram illustrating a deep neural network utilizing augmented documents for representation learning according to one embodiment of the present invention.
[0016] [Diagram 5] FIG. 5 is an exemplary processing system in which the method and system may be applied, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] According to an embodiment of the present invention, a contrastive learning approach using data augmentation techniques for learning document representations in an unsupervised manner is provided. Data augmentation is a technique for generating additional samples of relatively lower quality than the original data. The amount and variety of additional samples has shown effectiveness in various learning algorithms in computer vision and speech. Data augmentation is a technique for generating new and realistic training data of relatively lower quality than the original data points by applying transformations such as rotating and / or blurring images or synonym replacement of words in the text. For example, an image of an animal or vehicle can be adjusted to appear from a different angle, from a distance, or partially occluded, or the word "large" in the original document can be replaced with the words big, huge, substantial, and / or not small in one or more augmented documents. In this way, a small set of documents on a particular subject can be augmented for training a neural network without substantially changing the context and meaning of the documents.
[0018] In various embodiments, systems and methods are provided for unsupervised document embedding tasks that can be used to efficiently encode documents into compact vectors and train encoders that can be used for a variety of downstream tasks. The underlying meaning of a document is expressed in part by the words that appear in it. Some words in a document can be replaced, deleted, or inserted without changing the meaning or labeling information of the document.
[0019] Capturing the meaning of documents to obtain machine-understandable representations has a significant impact on various natural language processing (NLP) tasks. In one embodiment, document embeddings using contrastive augmentation are provided. Compared to the suboptimal baseline, classification error rate is reduced by up to 6.4% and classification performance is improved by up to 7.6%. Surprisingly, in classification tasks, the DECA method can match or even exceed fully supervised methods. High-quality document embeddings need to be invariant to various paraphrases that preserve the meaning of the original document.
[0020] In various embodiments, to address the data scarcity issue, contrastive learning with different augmentations for learning document representations can be used. A contrastive learning approach using data augmentation techniques learns document representations in an unsupervised manner. Data augmentation can be employed to include more information by generating new documents that maintain the same or similar meaning.
[0021] Doc2vecC computes document embeddings by simply averaging the embeddings of all words in a document. Dov2vec can learn document embeddings with contextual word predictions. The document embedding matrix can be kept in memory and is optimized together with the word embeddings.
[0022] A function that maps a document Di to a compact representation that preserves its semantics. The document embedding can be invariant to various paraphrases that preserve the meaning of the original document.
[0023] Contrastive learning is a framework for learning similar / dissimilar representations from data based on similarity measures and a contrastive loss function.
[0024] The contrastive loss includes the contrastive loss as a regularizer, which is the encoder loss l d Given a batch of N documents,
[0025] The expansion strategy can use two expansion methods to obtain a diversely represented document: thesaurus-based replacement and back-translation. In various embodiments, only words in the vocabulary are considered as replacement candidates. The thesaurus may contain a list of synonyms and antonyms for each word in the document.
[0026] Doc2vecC computes document embeddings by simply averaging the embeddings of all words in a document. Word-level operations such as synonym replacement to generate realistic and probabilistic augmentation examples work much better than augmentation at other granularities such as sentence-level or document-level ones. Learning document representations allows obtaining low-level original embeddings that preserve the semantic meaning of the document.
[0027] BERT stacks Transformer layers, including a self-attention sublayer and a feed-forward sublayer, to encode tokens in the input sequence.
[0028]
[0023] Referring now in detail to the drawing, FIG. 1, where like numerals represent the same or similar elements, there is illustrated an exemplary high level system / method for utilizing augmented documents for representation learning in accordance with one embodiment of the present invention.
[0029] Deep learning-based methods are available for long-text NLP tasks. However, the quality of representations obtained by existing methods is heavily affected by the problem of data scarcity, i.e., the lack of information when resources are scarce. Including more information can solve the resource scarcity problem. Data augmentation can generate additional samples from the original data points, which may be of relatively low quality. These generated additional training samples can improve the accuracy performance of deep learning methods, with one or more augmented documents being provided (e.g., fed) to another neural network and used for training. However, selecting an appropriate augmentation technique in an unsupervised setting without knowledge of any label information is not straightforward.
[0030] In one or more embodiments, the control document augmentation system 100 may have the augmented documents 130 fed into a document encoder 140 that generates document embeddings 150 for each of the negative documents 110, original documents 120, and input documents 110, 120, 130. For example, the negative document 110 of an original document of an image of a dog may contain an image of something other than a dog, while the positive instance of the augmented document 130 may contain, for example, a rotated or blurred image of a dog. Data augmentation can be employed to generate new documents that maintain the same or similar meaning, but contain more information.
[0031] The contrastive learning loss aims to maximize consistency under differently augmented views, allowing data-specific choices to impart the desired invariance.
[0032] In various embodiments, the document encoder 140
number
number
[0033] Including data augmentation in the contrast method can significantly improve the embedding quality in unsupervised document representation learning. Probabilistic augmentations generated by simple word-level operations perform much better than sentence-level and document-level augmentations.
[0034] In various embodiments, the document
number
number
[0035]
number
number
number
[0036]
number
number
[0037]
number
number
number
[0038]
number
number
number
number
[0039]
number
number
[0040]
number
number
[0041]
number
number
[0042] FIG. 2 is a block / flow diagram illustrating a document embedding system / method using contrastive expansion (DECA) according to one embodiment of the present invention.
[0043] In various embodiments, the probabilistic data augmentation generator 210 generates the probabilistic data augmentation
number
number
number
number
[0044] In various embodiments, the document encoder 140 performs the function
number
number
[0045]
number
[0046]
number
[0047] where U acts as the word embedding matrix and c t is the target word w in document D t where U is the local context of the target word w and V is a learnable projection matrix. To optimize U, Doc2vecC extends the Continuous Bag of Words (CBOW) model by treating documents as special tokens for the context, and t Maximize the probability of
[0048] The element-wise loss function of Doc2vecC is
[0049]
number
[0050] Here, the total loss is
number
[0051] The contrastive loss is introduced as a regularizer, and the encoder loss
number
[0052] Each document
number
number
number
number
number
number
[0053] The sample-wise control losses are as follows:
[0054]
number
[0055] Where:
number
number
[0056] The total loss is
number
[0057] Contrastive loss as a regularization term
number
[0058] The loss calculator 230 calculates the control loss
number
[0059] Within the SimSaim framework, a predictive MLP with batch normalization is first applied to obtain the output vector
number
number
[0060] The function D(·;) is the negative cosine similarity.
[0061]
number
[0062] Here, λ is a hyperparameter for setting the tradeoff between the two loss components.
[0063] When BERT is adopted as the backbone, the symmetric loss
number
[0064] Generating realistic augmented examples that preserve the meaning of the original document in an efficient manner is nontrivial. An input document can be paraphrased by replacing words based on synonyms, antonyms including negation prefixes, or their frequencies, while preserving its meaning. In synonym replacement, for each word, we first extract a set of replacement candidates using WordNet Synsets and remove out-of-vocabulary and infrequent candidates. For efficient computation, we also include the original word in the synonym set. To generate the augmented documents, for each word, we randomly select a word from the set of replacement candidates.
[0065] In the case of negative antonym replacement, an adjective or verb can be replaced with an antonym that includes a negative prefix such as "non."
[0066] In the case of uninformative word replacement, low frequency words can be replaced with high frequency synonyms.
[0067] The underlying meaning of a document may only be partially expressed by the document itself.
[0068] Back-translation is done by first translating document D from its original language (English in this study) to another language, such as German or French, to obtain D'. Then, document D' is translated into an augmented document D * The document-level back-translation can generate highly diverse paraphrases while preserving the meaning.
[0069] A wide range of document corpora are employed, including sentiment analysis (MR, IMDB), news classification (R8, R52, 20news), medical literature (Ohsumed), etc.
[0070] The embedding dimension is set to 100, except for the transformer-based model, which has an output dimension of 768. For each dataset, we first learn the respective embeddings using all documents. These document embeddings are then evaluated on two downstream tasks: linear classification and clustering.
[0071] Logistic regression is employed as the classifier and the test error rate is used as the evaluation criterion.
[0072] The data augmentation used in DECA generates new documents of relatively low quality, enriching the diversity of the text dataset and addressing the resource scarcity problem. DECA is also more robust to noise introduced into the augmented text. This allows DECA more flexibility in choosing different augmentation methods, resulting in higher quality embeddings. The newly generated documents and the original documents can be used to train the neural network.
[0073] A neural network is a generalized system that improves its functionality and accuracy through exposure to additional experimental data. A neural network is trained by exposure to experimental data. During training, the neural network stores and adjusts a number of weights that are applied to the input experimental data. By applying the adjusted weights to the data, the data can be identified as belonging to a particular predefined class from a set of classes, or the probability that the input data belongs to each class can be output.
[0074] Experimental data, also known as training data, from a set of cases is formatted as a string of values and fed to the input of the neural network. Each case can be associated with a known outcome or output. Each case can be represented as an (x,y) pair, where x represents the input data and y represents the known output. The input data can include a variety of different data types and can include multiple different values. The network can have one input node for each value that makes up the input data of the case, and a separate weight can be applied to each input value. The input data can be formatted, for example, as a vector, array, or string, depending on the architecture of the neural network to be built and trained.
[0075] Neural networks "learn" by comparing the neural network's output, generated from input data, with known values of the examples and adjusting stored weights to minimize the difference between the value of the output and the known values. Adjustments can be made to the stored weights through backpropagation. In this case, the effect of the weights on the value of the output can be determined by calculating the mathematical gradient and adjusting the weights to shift the output to the smallest difference. This optimization, called a gradient descent approach, is one non-limiting example of how training can be performed. A subset of examples with known values that were not used in training can be used to test and validate the accuracy of the neural network.
[0076] In operation, the trained neural network can be used on new data not previously used in training or validation by generalization. The adjusted weights of the neural network can be applied to the new data. The weights estimate a function formulated from the training examples. The parameters of the estimated function obtained by the weights are based on statistical inference.
[0077] For example, if a neural network is intended to predict the nature of the next input from the previous input data, the neural network can be configured to capture the time evolution of the input data. This can be achieved by introducing a time delay between inputting each subsequent data value. This can provide a short-term memory for the input data by exposing the nodes to the input data in a sequence where the data itself has its own time sequence.
[0078] The memory of a neural network can be increased by feeding the output generated by a node back as an input with a time delay, so that earlier input data influences the output of later input data, although the influence of earlier data can quickly decay.
[0079] FIG. 3 is a block / flow diagram illustrating a neural network for utilizing augmented documents for representation learning, according to one embodiment of the present invention.
[0080] In a hierarchical neural network, multiple nodes are arranged in layers. An exemplary simple neural network has an input layer 1020 of source nodes 1022 and a single computational layer 1030 with one or more computational nodes 1032 that also function as output nodes, where there is a single computational node 1032 for each category into which the input examples are considered to fall. The input layer 1020 may have a number of source nodes 1022 equal to the number of data values 1012 in the input data 1010. The data values 1012 of the input data 1010 may be represented as column vectors. Each computational node 1032 in the computational layer 1030 generates a linear combination of weight values from the input data 1010 provided to the input node 1020 and applies a differentiable nonlinear activation function to the sum. An exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).
[0081] FIG. 4 is a block / flow diagram illustrating a deep neural network for utilizing augmented documents for representation learning, according to one embodiment of the present invention.
[0082] A deep neural network, such as a multi-layer perceptron, has an input layer 1020 of source nodes 1022, one or more computational layers 1030 with one or more computational nodes 1032, and an output layer 1040 with a single output node 1042 for each category into which the input examples may be classified. The input layer 1020 may have a number of source nodes 1022 equal to the number of data values 1012 in the input data 1010. The computational nodes 1032 in the computational layer 1030 are also called hidden layers, since they are between the source nodes 1022 and the output nodes 1042 and are not directly observed. Each node 1032, 1042 in the computational layer generates a linear combination of weight values from the values output by the nodes in the previous layer and applies a nonlinear activation function that is differentiable over the range of the linear combination. The weights applied to the values from each previous node may be, for example, w 1 , w 2 , ..., w n-1 , w n The output layer provides the overall response of the network to the input data. Deep neural networks may be fully connected, in which case each node in a computational layer is connected to every other node in the previous layer, or they may have other configurations of connections between layers. If links between nodes are missing, the network is considered to be partially connected.
[0083] Training a deep neural network involves two stages: a forward stage, where the weights of each node are fixed and the inputs are propagated through the network, and a backward stage, where the error values are propagated backwards through the network to update the weight values. The parameters U, V can be updated by backpropagation.
[0084] Computational nodes 1032 in one or more computational (hidden) layers 1030 perform non-linear transformations on the input data 1012 that generate a feature space. Classes or categories are more easily separable in the feature space than in the original data space.
[0085] FIG. 5 is an exemplary processing system in which the method and system may be applied, according to one embodiment of the present invention.
[0086] The processing system 500 may include at least one processor (CPU) 504 and may have a graphics processing unit (GPU) 505 capable of performing vector calculations / manipulations operably connected to other components via a system bus 502. Also operably connected to the system bus 502 are a cache 506, a read only memory (ROM) 508, a random access memory (RAM) 510, an input / output (I / O) adapter 520, a sound adapter 530, a network adapter 540, a user interface adapter 550, and / or a display adapter 560.
[0087] The first storage device 522 and the second storage device 524 are operatively connected to the system bus 502 by an I / O adapter 520, where the recurrent neural network for generating the augmented document is stored to implement the functionality described herein. The storage devices 522 and 524 may be disk storage devices (e.g., magnetic or optical disk storage devices), solid-state storage devices, magnetic storage devices, etc. The storage devices 522 and 524 may be the same type of storage device or different types of storage devices. The contrastive document augmentation system 100 is stored in the storage device 524 and implemented by at least one processor (CPU) 504 and / or graphics processing unit (GPU) 505.
[0088] Speakers 532 are operably connected to the system bus 502 by a sound adapter 530. Transceiver 542 is operably connected to the system bus 502 by a network adapter 540. A display device 562 is operably connected to the system bus 502 by a display adapter 560.
[0089] A first user input device 552, a second user input device 554, and a third user input device 556 are operatively connected to the system bus 502 by a user interface adapter 550. The user input devices 552, 554, and 556 may be any of a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, or a device incorporating the functionality of at least two of these devices. Of course, other types of input devices may be used while maintaining the spirit of the principles of the present invention. The user input devices 552, 554, and 556 may be the same type of user input device or different types of user input devices. The user input devices 552, 554, and 556 are used to input information to the processing system 500 and output information from the processing system 500.
[0090] In various embodiments, the processing system 500 may include other elements (not shown) or omit certain elements, as would be readily apparent to one of ordinary skill in the art. For example, the processing system 500 may include various other types of input and / or output devices, depending on the particular implementation, as would be readily apparent to one of ordinary skill in the art. For example, various wireless and / or wired input and / or output devices may be used. Furthermore, various configurations of additional processors, controllers, memory, etc. may be used, as would be readily apparent to one of ordinary skill in the art. These and other variations of the processing system 500 will be readily apparent to one of ordinary skill in the art from the teachings of the present principles provided herein.
[0091] It should further be understood that the processing system 500 is a system for performing each of the embodiments of the present method / system. Part or all of the processing system 500 may be implemented with one or more of the elements of Figures 1 and 2. It should further be understood that the processing system 1000 can perform at least a portion of the methods described herein, including, for example, at least a portion of the methods of Figures 1 and 2.
[0092] The embodiments described herein may be implemented entirely in hardware, entirely in software, or contain both hardware and software elements, in a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.
[0093] An embodiment may include a computer program product accessible from a computer usable or computer readable medium that provides program code for use by or in connection with a computer or any instruction execution system. A computer usable or computer readable medium may include any device that stores, conveys, propagates, or transfers a program for use by or in connection with an instruction execution system, device, or apparatus. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or device or apparatus), or a propagation medium. The medium may include computer readable media such as semiconductor or solid state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disk, and optical disk.
[0094] Each computer program is stored in a machine-readable storage medium or device (e.g., a program memory or a magnetic disk) that can be read by a general-purpose or special-purpose programmable computer. The computer program is read from the storage medium or device by a computer that executes the procedures described herein for configuring and controlling operations of the computer. The system of the present invention also contemplates a computer-readable storage medium containing a computer program configured to cause a computer to operate in a specific and predefined manner to perform the functions described herein.
[0095] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory, bulk memory devices, and cache memory for temporarily storing at least some program code used during the actual execution of the program code to reduce the number of times code is retrieved from bulk memory devices during execution of a process. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be connected to the system either directly or through I / O controllers.
[0096] Network adapters may be coupled with the data processing system to enable the system to be coupled to other data processing systems or remote printers or memory devices through private or public networks. Modems, cable modems and Ethernet cards are just a few of the currently available types of network adapters.
[0097] As used herein, the term "hardware processor subsystem" or "hardware processor" may refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution units, etc.). The one or more data processing elements may include a central processing unit, a graphics processing unit, and / or individual processors or computing element-based controllers (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.). In any embodiment, the hardware processor subsystem may include one or more memories that may be on-board or off-board, or may be dedicated for use with the hardware processor subsystem (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.).
[0098] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code for achieving a particular result.
[0099] In other embodiments, the hardware processor subsystem may include dedicated circuitry to perform one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).
[0100] These and other variations of the hardware processor subsystem are also contemplated by embodiments of the present invention.
[0101] Reference herein to "one embodiment" or "one embodiment," as well as other variations, of the invention means that a particular feature, configuration, characteristic, etc. described in connection with an embodiment is included in at least one embodiment of the invention. Thus, appearances of the phrases "in one embodiment" or "in an embodiment," as well as any other variations appearing in various places throughout this specification, are not necessarily all referring to the same embodiment. However, it should be understood that given the teachings of the invention provided herein, features of one or more embodiments may be combined.
[0102] For example, the use of any of "A / B," "A and / or B," and "at least one of A and B" should be understood to be intended to include the selection of only the first listed option (A), the selection of only the second listed option (B), or the selection of both options (A and B). By way of further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to include the selection of only the first listed option (A), the selection of only the second listed option (B), the selection of only the third listed option (C), the selection of only the first and second listed options (A and B), the selection of only the first and third listed options (A and C), the selection of only the second and third listed options (B and C), or the selection of all three options (A, B, and C). The above examples may be expanded depending on the number of items listed, as would be readily apparent to one of ordinary skill in the art.
[0103] The foregoing should be understood in all respects as illustrative and exemplary, and not restrictive, and the scope of the invention disclosed herein should be determined not from the detailed description, but from the claims which are to be interpreted in accordance with the broadest possible interpretation permitted by the Patent Law. It should be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that various modifications may be made by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Although aspects of the invention have been described above with the fine detail and particularity required by the Patent Law, the scope of the claims which are sought to be protected by Letters Patent are set forth in the appended claims.
Claims
1. 1. A method for expanding a dataset, comprising: The computer providing an original document (120) to said computer-implemented data augmentation generator (210) to generate one or more augmented documents (220); Calculating a contrast loss (230) between the original document (120) and the one or more augmented documents (220); training (1030) a neural network using the original document (120) and the one or more augmented documents (220); generating at least one of the one or more augmented documents by computing an embedding of the original document using Doc2vecC and computing a contrastive loss of the embedded document.
2. The method of claim 1 , wherein at least one of the one or more augmented documents is generated by replacing words in the original document with synonyms.
3. The method of claim 1 , wherein at least one of the one or more augmented documents is generated by replacing words in the original document with their antonyms comprising a negation prefix before the antonym.
4. The method of claim 1 , wherein at least one of the one or more augmented documents is generated by rotating and / or blurring a digital image.
5. The contrast loss is [0010] The method of claim 1 , wherein the calculated
6. The sum of the control losses is [0025] The method of claim 5, wherein the eigenvalue is calculated using:
7. 1. A system for augmenting a dataset, comprising: One or more processors (504, 505); a memory (510, 522) operatively connected to said one or more processors (504, 505); a data augmentation generator (210) stored in said memory and configured to generate one or more augmented documents (220) from an original document (120); a loss calculator (230) configured to calculate a contrast loss between the original document (120) and the one or more augmented documents (220); having the data augmentation generator is further configured to generate at least one of the one or more augmented documents by computing an embedding of the original document using Doc2vecC and computing a contrast loss for the embedded document.
8. The system of claim 7 , wherein the data augmentation generator is further configured to generate at least one of the one or more augmented documents by replacing words in the original document with synonyms.
9. 8. The system of claim 7, wherein the data augmentation generator is further configured to generate at least one of the one or more augmented documents by replacing words in the original document with antonyms comprising a negation prefix before the antonyms.
10. 8. The system of claim 7, wherein the data augmentation generator is further configured to generate at least one of the one or more augmented documents by rotating and / or blurring a digital image of at least one of the one or more augmented documents.
11. The contrast loss is [0030] The system of claim 7, wherein the calculation is performed using:
12. The sum of the control losses is [0045] The system of claim 11 , wherein the calculation is performed using
13. A program for causing a computer to execute a procedure for expanding a dataset, comprising: The computer includes: receiving an original document (120) into said computer-implemented data augmentation generator (210) to generate one or more augmented documents (220); computing a contrast loss (230) between the original document (120) and the one or more augmented documents (220); training (1030) a neural network using the original document (120) and the one or more augmented documents (220); Run the command, 11. A program for generating at least one of the one or more augmented documents by computing an embedding of the original document using Doc2vecC and computing a contrastive loss of the embedded document.
14. 14. The program product of claim 13, for generating at least one of the one or more augmented documents by replacing words in the original document with synonyms.
15. 14. The program of claim 13, for generating at least one of the one or more augmented documents by replacing words in the original document with antonyms comprising a negation prefix before the antonym.
16. 14. The program of claim 13, for generating at least one of the one or more augmented documents by rotating and / or blurring a digital image.
17. The control loss is [0050] Calculate using The total of the control losses is [006] The program according to claim 13, for causing calculation using
Citation Information
Patent Citations
Recommendation reason screening method and device and electronic equipment
CN111831902A
Learning data extension device, learning device, translation device, and program
JP2020126360A
Method of training a neural network
US20180211164A1
Training a neural network using augmented training datasets
US20190130218A1
Information processing device and information processing method
WO2020170593A1
