A pre-training model based on continuous and discrete mixed representation and a training method thereof

Abstract

The application discloses a pre-training model based on continuous and discrete mixed representation and a training method thereof. The pre-training model is composed of multiple novel Transformer architectures supporting continuous and discrete representation, and the architecture comprises: a continuous semantic representation acquisition module, which is used for acquiring continuous semantic representation; a discrete semantic representation acquisition module, which is used for acquiring discrete semantic representation; a fusion module, which is used for fusing the continuous semantic representation and the discrete semantic representation to obtain mixed semantic representation; and an output module, which is used for outputting the three kinds of semantic representation. The application can solve the problem that the existing pre-training model is sensitive to data noise; reduce the parameter space of a large-scale pre-training model through discrete representation, provide a new idea for the prototype design of the large-scale pre-training model; meanwhile, the application also provides a new direction for the application of the large-scale pre-training model, outputs a structured knowledge graph representation, increases the explainability of the model, and promotes the development of logical reasoning based on discrete representation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a pre-trained model based on a hybrid representation of continuous and discrete representations and its training method. Background Technology

[0002] Pre-trained models are a hot topic in the field of artificial intelligence. They aim to pre-train models using massive amounts of text data and quickly transfer the resulting models to downstream tasks. With the successive introduction of large-scale pre-trained models such as BERT, GPT3, and T5, pre-trained model technology has shone brightly in artificial intelligence fields such as natural language processing and computer vision, achieving excellent results. Although current pre-trained models can achieve good results on a range of downstream tasks, they can only use continuous semantic representations to characterize information in text data within a continuous semantic space. This results in poor robustness to data noise and makes it difficult to endow the models with cognitive abilities such as reasoning and decision-making. Conversely, since language itself is discrete, discrete semantic representations are naturally suitable for complex reasoning and decision-making learning, and discrete representations have good robustness to noisy data. However, due to problems such as sparse encoding and non-differentiable discrete variables, building pre-trained models based on discrete semantic representations is extremely challenging. Therefore, how to build pre-trained models based on discrete semantic representations and how to comprehensively utilize the advantages of continuous and discrete semantic representations are important ways to improve the cognitive abilities of models in reasoning, decision-making, etc. Summary of the Invention

[0003] To address the aforementioned problems, this invention aims to provide a pre-trained model based on a hybrid representation of continuous and discrete representations, and its training method.

[0004] The technical solution of the present invention is as follows:

[0005] On the one hand, a pre-trained model based on a hybrid continuous and discrete representation is provided, consisting of multiple novel Transformer architectures supporting both continuous and discrete representations. The novel Transformer architectures include:

[0006] The continuous semantic representation acquisition module is used to acquire continuous semantic representations;

[0007] The discrete semantic representation acquisition module is used to acquire discrete semantic representations.

[0008] A fusion module is used to fuse the continuous semantic representation with the discrete semantic representation to obtain a hybrid semantic representation;

[0009] The output module is used to output the continuous semantic representation, the discrete semantic representation, and the hybrid semantic representation.

[0010] Preferably, the discrete semantic representation acquisition module includes a decomposition module, a discretization module, and a combination module;

[0011] The decomposition module is used to decompose the continuous semantic representation into multiple semantic primitive representations;

[0012] The discretization module is used to discretize the multiple semantic primitive representations to obtain the discretized semantic primitive representations.

[0013] The combination module is used to perform weighted combination of the discretized semantic primitive representations to obtain discrete semantic representations.

[0014] Preferably, the continuous semantic representation acquisition module uses the traditional Transformer architecture to acquire the continuous semantic representation of the input word.

[0015] Preferably, the decomposition module dynamically acquires semantic primitive representations based on a sparse attention mechanism. The number of semantic primitive representations is automatically determined by the output of the sparse attention mechanism, and the semantic primitive representations corresponding to non-zero attention scores are automatically selected.

[0016] Preferably, the discretization module uses a vector quantization variational autoencoder to map each input primitive representation into a discrete representation.

[0017] Preferably, the combination module uses an attention mechanism to achieve a weighted combination of the discretized semantic primitive representations.

[0018] Preferably, the fusion module first uses a self-attention mechanism, and then performs mean fusion of continuous semantic representation and discrete semantic representation.

[0019] On the other hand, a training method for a pre-trained model based on a hybrid representation of continuous and discrete representations is also provided, which learns the parameters of the pre-trained model through the following pre-training tasks:

[0020] The next sentence prediction task is used to teach the pre-trained model the logical relationships between sentences.

[0021] The masked word prediction task is used to enable pre-trained models to learn semantic relationships between words;

[0022] Controlled continuous semantic decomposition task, used to enable pre-trained models to learn the correct mapping relationship between the continuous semantic representation of a word and its semantic primitive representation;

[0023] The semantic composition task is used to enable pre-trained models to learn semantic invariance before and after discrete composition operations.

[0024] As a preferred approach, the optimization objectives during the learning process include:

[0025] (1) Targeting hybrid semantic representation

[0026] a) Minimize its loss on the next sentence prediction task.

[0027] b) Minimize its loss on the mask word prediction task.

[0028] (2) Targeting continuous semantic representation

[0029] a) Minimize its loss on the next sentence prediction task.

[0030] b) Minimize its loss on the mask word prediction task.

[0031] (3) Targeting discrete semantic representation

[0032] a) Minimize its loss on the next sentence prediction task.

[0033] b) Minimize its loss on the mask word prediction task.

[0034] c) Loss of semantic composition task

[0035] i. Minimizing the vector quantization loss and information input loss between discrete and continuous semantic representations in a vector quantization variational autoencoder.

[0036] ii. Maximize the difference between discrete and continuous semantic representations as the loss in the next sentence prediction task.

[0037] iii. Maximizing the difference between discrete and continuous semantic representations as the loss in the masked word prediction task.

[0038] iv. Maximize the entropy of the difference between discrete semantic representations and continuous semantic representations

[0039] d) Loss of controlled continuous semantic decomposition task

[0040] i. Minimize the mismatch rate between the decomposed semantic primitive representations and the true semantic primitives.

[0041] Based on the above loss function, and by training with text data, the pre-trained model based on hybrid representation can be obtained.

[0042] The beneficial effects of this invention are:

[0043] This invention addresses the problem of existing pre-trained models being sensitive to data noise; it reduces the parameter space of large-scale pre-trained models through discretization, providing a new approach for the prototype design of large-scale pre-trained models; at the same time, it also provides a new direction for the application of large-scale pre-trained models, outputting structured knowledge graph representations, increasing the interpretability of the model, and promoting the development of logical reasoning based on discrete representations. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the pre-trained model framework based on a hybrid representation of continuous and discrete representations of the present invention.

[0046] Figure 2 This is a schematic diagram of the novel Transformer architecture that supports both continuous and discrete representations according to the present invention.

[0047] Figure 3 This is a schematic diagram of the controlled continuous semantic decomposition task of the present invention;

[0048] Figure 4 This is a schematic diagram of the semantic combination task of the present invention. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and technical features described in this application can be combined with each other. It should also be pointed out that, unless otherwise indicated, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terms "comprising" or "including" and similar words used in this invention refer to elements or objects preceding the word that encompass the elements or objects listed following the word and their equivalents, without excluding other elements or objects.

[0050] On the one hand, such as Figure 1 As shown, this invention provides a pre-trained model based on a hybrid continuous and discrete representation, composed of multiple novel Transformer architectures supporting both continuous and discrete representations. The novel Transformer architectures are as follows: Figure 2 As shown, it includes:

[0051] (1) Continuous semantic representation acquisition module, used to acquire continuous semantic representation;

[0052] In one specific embodiment, the continuous semantic representation acquisition module uses a traditional Transformer architecture to acquire the continuous semantic representation of the input words. Given an input sentence, for each word in the sentence (the i-th word is denoted as x...),... i The input word x is obtained using the traditional Transformer architecture (i.e., the Transformer structure in traditional pre-trained models, which is existing technology, and the specific architecture will not be described in detail here).i The continuous semantic representation, denoted as h ct .

[0053] (2) Discrete semantic representation acquisition module, used to acquire discrete semantic representation;

[0054] The discrete semantic representation acquisition module includes a decomposition module, a discretization module, and a combination module; the decomposition module is used to decompose the continuous semantic representation into multiple semantic primitive representations; the discretization module is used to discretize the multiple semantic primitive representations to obtain discretized semantic primitive representations; the combination module is used to perform weighted combination of the discretized semantic primitive representations to obtain discrete semantic representations.

[0055] In one specific embodiment, the decomposition module dynamically acquires semantic primitive representations based on a sparse attention mechanism. The number of semantic primitive representations is automatically determined by the output of the sparse attention mechanism, and the semantic primitive representations corresponding to non-zero attention scores are automatically selected. Input h ct The decomposition module outputs several semantic primitive representations. Figure 1 In this context, there are a total of n = 9 semantic primitives, where the representation of the i-th semantic primitive is denoted as c. i The final output contains 3 semantic primitive representations.

[0056] In one specific embodiment, the discretization module uses a vector quantization variational autoencoder to map each input primitive representation into a discrete representation. Figure 1 In this context, a 3×3 matrix represents a discrete semantic vector space with 9 discrete representations, and the discretized semantic primitive representation is denoted as z. i .

[0057] In one specific embodiment, the combination module employs an attention mechanism to achieve a weighted combination of the discretized semantic primitive representations, denoted as h. cp .

[0058] (3) A fusion module, used to fuse the continuous semantic representation with the discrete semantic representation to obtain a hybrid semantic representation;

[0059] In one specific embodiment, the fusion module first uses a self-attention mechanism, and then performs mean fusion of the continuous semantic representation and the discrete semantic representation to obtain a hybrid semantic representation denoted as h.

[0060] (4) Output module, used to output the continuous semantic representation, the discrete semantic representation and the hybrid semantic representation.

[0061] On the other hand, the present invention also provides a training method for a pre-trained model based on a hybrid representation of continuous and discrete representations, which learns the parameters of the pre-trained model through the following pre-training tasks:

[0062] (1) NextSentencePrediction is a task that predicts whether the second sentence of the two input sentences has semantic continuity with the first sentence, that is, whether the second sentence is the next sentence of the first sentence at the semantic level. This is used to enable the pre-trained model to learn the logical relationship between sentences.

[0063] (2) Masked Token Prediction is a task that predicts what words are masked in an input sentence, which is used to help the pre-trained model learn the semantic relationships between words.

[0064] (3) Guided Decomposition Task, such as Figure 3 As shown, this represents the semantic primitives corresponding to a word in the prediction input, which is used to enable the pre-trained model to learn the correct mapping relationship between the continuous semantic representation of a word and the semantic primitive representation.

[0065] The controlled continuous semantic decomposition task is used to guide the model on how to decompose the semantic information of input words. The semantic information of a word can be represented by several semantic primitives (for example, the word "wife" in the diagram can be decomposed into three semantic primitives: family, spouse, and female. A semantic primitive is a concept in linguistics, used to characterize the smallest unit of semantics). However, not all words have known semantic primitives; in the Chinese domain, only a subset of words have known semantic primitives. Therefore, this invention uses a data-driven approach to find the semantic primitives of all words. That is, relying on the controlled continuous semantic decomposition task, any input word is decomposed into its semantic primitives, specifically by dynamically acquiring semantic primitive representations based on a sparse attention mechanism. Furthermore, the "controlled" aspect of the controlled continuous semantic decomposition task is reflected in using words whose semantic primitives are known to guide the model in learning parameters. This guidance is achieved by comparing the mismatch rate between the semantic primitives decomposed by the model and the true semantic primitives, and minimizing this mismatch rate (i.e., di in the optimization objective below) by establishing an optimization function, thereby achieving parameter learning.

[0066] (4) Semantic Composition Task, such as Figure 4 As shown, for an input word, the goal is to minimize the semantic difference between the representation after discretization and recombination of the semantic primitive representation and its original continuous representation, so that the pre-trained model can learn the semantic invariance before and after the discrete recombination operation.

[0067] The semantic composition task involves recombining discretized semantic primitive representations into words. This recombination operation is also achieved through a data-driven approach (i.e., an attention mechanism). The goal of this task is to utilize the semantic information h contained in the recombined words. cp , and the semantic information h of the word before recombination ct They are basically the same. The implementation method is by requiring h cp and h ct The difference has no semantic information. This difference has no effect on the next sentence prediction task and the mask word prediction task. Moreover, this difference has the largest information entropy (the largest information entropy means the least amount of information), which is the three parts of c.ii, c.iii and c.iv in the optimization objective below.

[0068] In one specific embodiment, the pre-trained model optimizes the following objectives during the learning process:

[0069] (1) Targeting hybrid semantic representation

[0070] a) Minimize its loss on the next sentence prediction task.

[0071] b) Minimize its loss on the mask word prediction task.

[0072] (2) Targeting continuous semantic representation

[0073] a) Minimize its loss on the next sentence prediction task.

[0074] b) Minimize its loss on the mask word prediction task.

[0075] (3) Targeting discrete semantic representation

[0076] a) Minimize its loss on the next sentence prediction task.

[0077] b) Minimize its loss on the mask word prediction task.

[0078] c) Loss of semantic composition task

[0079] i. Minimizing the vector quantization loss and information input loss between discrete and continuous semantic representations in a vector quantization variational autoencoder.

[0080] ii. Maximize the difference between discrete and continuous semantic representations as the loss in the next sentence prediction task.

[0081] iii. Maximizing the difference between discrete and continuous semantic representations as the loss in the masked word prediction task.

[0082] iv. Maximize the entropy of the difference between discrete semantic representations and continuous semantic representations

[0083] d) Loss of controlled continuous semantic decomposition task

[0084] i. Minimize the mismatch rate between the decomposed semantic primitive representations and the true semantic primitives.

[0085] Based on the above loss function, and by training with text data, the pre-trained model based on hybrid representation can be obtained.

[0086] In one specific embodiment, training is performed on large-scale text datasets such as Wikipedia, using eight NVIDIA V100 high-performance GPUs, and the parameter optimization algorithm employs Adam. The resulting pre-trained model yields...

[0087] 1. Continuous representation effect: The continuous semantic representation effect obtained by the present invention is on par with the semantic representation effect of traditional pre-trained models, indicating that the present invention can be applied to a wide range of downstream tasks, just like traditional pre-trained models.

[0088] 2. Discrete representation effect

[0089] a) Noise robustness: On noisy data, the performance of this invention is 10-20% better than that of traditional pre-trained models;

[0090] b) Generalization ability: On data with out-of-distribution generalization (i.e., this type of data does not appear in the training or fine-tuning dataset of the model), the performance of the present invention is improved by 14-30% compared with traditional pre-trained models;

[0091] c) Fine-tuning efficiency: At different proportions of fine-tuning data, the present invention requires less data to achieve the same effect as the traditional pre-trained model, with an average reduction of about 23% in data volume.

[0092] 3. Continuous + Discrete Representation Effect: This representation can effectively combine the advantages of continuous and discrete representations.

[0093] In summary, this invention addresses the problem of existing pre-trained models being sensitive to data noise. Compared with existing technologies, this invention represents a significant advancement.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A pre-trained model based on a hybrid representation of continuous and discrete representations, characterized in that, It consists of multiple novel Transformer architectures that support both continuous and discrete representations, and the novel Transformer architectures include: The continuous semantic representation acquisition module is used to acquire continuous semantic representations; A discrete semantic representation acquisition module is used to acquire discrete semantic representations; the discrete semantic representation acquisition module includes a decomposition module, a discretization module, and a combination module. The decomposition module is used to decompose the continuous semantic representation into multiple semantic primitive representations; the decomposition module dynamically obtains semantic primitive representations based on a sparse attention mechanism, and the number of semantic primitive representations is automatically determined by the output of the sparse attention mechanism, and the semantic primitive representations corresponding to non-zero attention scores will be automatically selected. The discretization module is used to discretize multiple semantic primitive representations to obtain discretized semantic primitive representations; the discretization module uses a vector quantization variational autoencoder to map each input semantic primitive representation into a discrete representation. The combination module is used to perform weighted combination of the discretized semantic primitive representations to obtain discrete semantic representations; the combination module uses an attention mechanism to realize the weighted combination of the discretized semantic primitive representations. A fusion module is used to fuse the continuous semantic representation with the discrete semantic representation to obtain a hybrid semantic representation; The output module is used to output the continuous semantic representation, the discrete semantic representation, and the hybrid semantic representation.

2. The pre-trained model based on a hybrid continuous and discrete representation according to claim 1, characterized in that, The continuous semantic representation acquisition module uses the traditional Transformer architecture to acquire the continuous semantic representation of the input word.

3. The pre-trained model based on a hybrid continuous and discrete representation according to claim 1 or 2, characterized in that, The fusion module first uses a self-attention mechanism, and then performs mean fusion of continuous semantic representation and discrete semantic representation.

4. The training method for a pre-trained model based on a hybrid continuous and discrete representation as described in any one of claims 1-3, characterized in that, The parameters of the pre-trained model are learned through the following pre-training tasks: The next sentence prediction task is used to teach the pre-trained model the logical relationships between sentences. The masked word prediction task is used to enable pre-trained models to learn semantic relationships between words; Controlled continuous semantic decomposition task, used to enable pre-trained models to learn the correct mapping relationship between the continuous semantic representation of a word and its semantic primitive representation; The semantic composition task is used to enable pre-trained models to learn semantic invariance before and after discrete composition operations.

5. The training method for the pre-trained model based on a hybrid continuous and discrete representation according to claim 4, characterized in that, During the learning process, optimization goals include: (1) For hybrid semantic representation a) Minimize its loss on the next sentence prediction task. b) Minimize its loss on the mask word prediction task. (2) For continuous semantic representation a) Minimize its loss on the next sentence prediction task. b) Minimize its loss on the mask word prediction task. (3) For discrete semantic representation a) Minimize its loss on the next sentence prediction task. b) Minimize its loss on the mask word prediction task. c) Loss of semantic composition task i. Minimizing the vector quantization loss and information input loss between discrete and continuous semantic representations in a vector quantization variational autoencoder. ii. Maximize the difference between discrete and continuous semantic representations as the loss in the next sentence prediction task. iii. Maximizing the difference between discrete and continuous semantic representations as the loss in the masked word prediction task. iv. Maximize the entropy of the difference between discrete semantic representations and continuous semantic representations d) Loss of controlled continuous semantic decomposition task i. Minimize the mismatch rate between the decomposed semantic primitive representations and the true semantic primitives. Based on the above loss function, and by training with text data, a pre-trained model based on hybrid representation can be obtained.

Images