A model pre-training method based on masked autoencoder and noise enhancement

Abstract

The present invention provides a model pre-training method based on masked autoencoders and noise enhancement, which relates to the technical field of natural language processing. The method includes constructing an asymmetric encoding-decoding model, and improving the diversity of training signals through differentiated mask ratios and decoding mechanisms. Then, a noise injection mechanism is introduced to enhance the robustness of the model against disturbances by adding noise to the embedding, and two improvements are proposed: one is to dynamically adjust the noise amplitude, using larger noise to enhance robustness in the early stage of training, and reducing noise to improve accuracy in the later stage; the other is to use KL divergence to guide noise generation in the later stage of training, measure the distribution difference between the original embedding and the noisy embedding, and make the noise more intelligent for the weaknesses of the model. Pre-training and evaluation are carried out on multiple datasets, significantly improving the dense retrieval performance in zero-sample and supervised learning scenarios. Ultimately, this method can improve the accuracy and stability of sentence representation in retrieval tasks without additional fine-tuning of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a model pre-training method based on masked autoencoder and noise enhancement. Background Art

[0002] In existing dense search and natural language processing systems, traditional approaches rely on lexical retrieval techniques, such as TF-IDF and BM25, which determine relevance by calculating word frequency statistics or probability scores between queries and documents. These methods have shown some effectiveness in simple text matching tasks, particularly in keyword-driven search scenarios. However, their primary focus on surface-level lexical matching makes it difficult to capture the deeper semantic connections between queries and documents. This is particularly true when dealing with complex sentences or cross-document information, often failing to meet users' demands for more sophisticated semantic understanding. With the advancement of deep learning technology, embedding-based language models (such as BERT and RoBERTa) have been introduced to the dense search field. By converting sentences into vector representations and calculating semantic similarity, they significantly improve understanding of contextual semantics. These models can identify synonyms and contextual dependencies, surpassing the limitations of traditional methods. However, despite their advancements in semantic search, embedding models still face several key challenges.

[0003] First, existing embedding models mainly rely on general semantic representations in the pre-training stage in dense retrieval tasks. Although they can capture sentence-level semantic information, they are not robust enough when facing input perturbations (such as spelling errors, word order changes, or noisy data). This vulnerability causes the model to produce unstable retrieval results due to data variation in actual application scenarios, especially in large-scale real-world retrieval tasks. Secondly, traditional pre-training methods usually optimize the model based on label-level tasks (such as masked language modeling). The generated sentence representations are often not targeted in dense retrieval tasks and cannot fully adapt to the retrieval scenarios' requirements for sentence-level semantic discrimination. In addition, existing embedding models usually do not consider the training signals of adversarial perturbations during training, resulting in their performance degradation in complex or noisy environments. For example, when processing diverse queries on datasets such as MSMARCO, the model may not be able to effectively distinguish subtle semantic differences.

[0004] In addition, in order to improve the performance of the model in specific retrieval tasks, existing methods often require frequent fine-tuning of the pre-trained model to adapt to different data sets or domain requirements. This fine-tuning process not only consumes a lot of computing resources, but also increases the complexity of deployment and is difficult to maintain high efficiency in large-scale systems. Even if the model performance is improved through fine-tuning, when faced with incomplete or noisy inputs, the model may still generate inaccurate embedding representations, affecting the quality of retrieval results. The traditional pre-training paradigm lacks joint optimization of robustness and semantic reconstruction, which limits the model's migration ability in zero-shot or supervised learning scenarios, making it difficult to directly meet the requirements of intensive retrieval tasks for high-quality sentence representations. Therefore, how to improve the robustness and semantic representation capabilities of the model in the pre-training stage has become a technical problem that needs to be solved urgently in the current field of intensive retrieval. Summary of the Invention

[0005] To this end, an embodiment of the present invention provides a model pre-training method and system based on masked autoencoder and noise enhancement, which is used to solve the problem of insufficient sentence-level representation capability of pre-trained models in intensive retrieval tasks in the prior art.

[0006] In order to solve the above problems, an embodiment of the present invention provides a model pre-training method based on a masked autoencoder and noise enhancement, which includes:

[0007] S1: Build an asymmetric encoder-decoder model, where the encoder is a full-scale deep neural network for generating an embedded representation of the input sentence, and the decoder is a lightweight single-layer neural network for reconstructing the input sentence;

[0008] S2: Apply a differentiated masking strategy to the input sentence, where the encoder mask ratio is 15%-30% and the decoder mask ratio is 50%-75% to generate differentiated training signals;

[0009] S3: A dynamic noise injection mechanism is introduced during the encoder embedding generation process. By adding noise of different amplitudes at different stages of the training process, the robustness of the asymmetric encoding-decoding model against perturbations is enhanced.

[0010] S4: Use KL divergence to guide noise generation and optimize the noise generation direction by calculating the distribution difference between the original embedding and the noisy embedding;

[0011] S5: Pre-train the asymmetric encoding-decoding model using multiple datasets and optimize the parameters of the asymmetric encoding-decoding model through a joint loss of semantic reconstruction and noise adversarial loss;

[0012] S6: The pre-trained asymmetric encoder-decoder model is directly applied to dense retrieval tasks in zero-shot or supervised learning scenarios to output a vector representation of the sentence.

[0013] Preferably, in the asymmetric encoding-decoding model, the encoder adopts a multi-layer Transformer architecture to output a fixed-dimensional embedding vector, and the decoder adopts a single-layer Transformer architecture, the number of its parameters does not exceed 1 / 10 of the number of parameters of the encoder, and the parameters of the encoder and decoder are initialized through a pre-trained language model.

[0014] Preferably, step S2 specifically includes:

[0015] For the encoder mask operation, first determine the mask ratio , the range is , calculate the number of words that need to be masked , then from the sentence position Randomly select positions, denoted as a set ;for Each position in , the word Replace with special token [MASK] to generate masked sentences , which is of the form , or , ; This mask sentence As encoder input, used to generate embedding and the masked word vectors ;

[0016] In contrast, the decoder masking operation uses a higher mask ratio , ranging from 50% to 75%, calculate the number of masked words and from Randomly select positions, denoted as a set , May contain ; Similarly, Words in Replace with special mark , generate masked sentences , , or , , and as decoder input and position encoding Together they are used to reconstruct the original sentence ;

[0017] in, represents the floor function, represents the length of the sentence, Encoder(·) is the encoder function, Decoder function.

[0018] Preferably, the dynamic noise injection mechanism dynamically adjusts the noise amplitude according to the training progress, and its calculation formula is:

[0019] ;

[0020] in, represents the dynamic noise amplitude, represents the initial noise amplitude, Indicates the attenuation rate, which is used to control the speed of amplitude decrease. represents the total number of training steps, Indicates the current training step number, Indicates the minimum amplitude of noise.

[0021] Preferably, in the KL divergence-guided noise generation, the noise direction 𝜂 is calculated by the following formula:

[0022] ;

[0023] in, Represents the gradient under Gaussian approximation, pointing to the direction of the largest distribution difference, represents the gradient operation on the noise vector N, represents the KL divergence, Indicates the dynamic noise amplitude; is used to measure the difference between two embedding distributions, and The original embedding and noisy embedding The probability distribution of is a random vector sampled from the unit sphere, providing an exploratory perturbation; is the dynamic weight, ranging from , Indicates L2 norm normalization.

[0024] Preferably, the noise embedding Expressed as:

[0025] ;

[0026] in, represents the dynamic noise amplitude, is the optimal noise direction under the influence of KL divergence, is the regularization coefficient, which is adjusted through experiments. represents the original embedding The standard deviation of represents the minimum function.

[0027] Preferably, the semantic reconstruction and noise adversarial joint loss is:

[0028] ;

[0029] in, For semantic reconstruction and noise adversarial joint loss, and are weight coefficients ranging from 0 to 1. is the semantic reconstruction loss, for noise combating losses.

[0030] Preferably, the semantic reconstruction loss is:

[0031] ;

[0032] in, For the mask sentence, are decoder parameters, is the sentence length, Indicates that the decoder is given a mask sentence and decoder parameters Under the condition of The word is probability.

[0033] Preferably, the noise resistance loss is:

[0034] ;

[0035] in, represents a masked sentence, represents the masked sentence after adding dynamic noise, represents the square of the L2 norm, and Encoder(·) is the encoder function.

[0036] The embodiment of the present invention further provides a model pre-training system based on a masked autoencoder and noise enhancement, which is used to implement the above-mentioned model pre-training method based on a masked autoencoder and noise enhancement, specifically comprising:

[0037] Asymmetric encoding-decoding model building module, used to build an asymmetric encoding-decoding model, where the encoder is a full-scale deep neural network for generating embedded representations of input sentences, and the decoder is a lightweight single-layer neural network for reconstructing the input sentences;

[0038] Differentiated masking strategy module, which applies a differentiated masking strategy to the input sentence, with an encoder masking ratio of 15%-30% and a decoder masking ratio of 50%-75% to generate differentiated training signals;

[0039] Dynamic noise injection module, which is used to introduce a dynamic noise injection mechanism during the encoder embedding generation process. By adding noise of different amplitudes at different stages of the training process, it enhances the robustness of the asymmetric encoding-decoding model against perturbations.

[0040] The KL divergence-guided noise generation module is used to use KL divergence to guide noise generation and optimize the noise generation direction by calculating the distribution difference between the original embedding and the noisy embedding;

[0041] Model pre-training and parameter optimization module, which is used to pre-train the asymmetric encoding-decoding model using multiple datasets and optimize the parameters of the asymmetric encoding-decoding model through semantic reconstruction and noise adversarial joint loss;

[0042] The model application module is used to directly apply the pre-trained asymmetric encoding-decoding model to dense retrieval tasks in zero-shot or supervised learning scenarios, and output the vector representation of the sentence.

[0043] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned model pre-training method based on masked autoencoder and noise enhancement.

[0044] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned model pre-training method based on masked autoencoder and noise enhancement.

[0045] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0046] (1) Dual improvement in robustness and semantic representation capabilities: This paper significantly enhances the model's ability to resist input perturbations through a dynamic noise injection mechanism and adversarial training, thereby improving the robustness of sentence representation. By adopting an asymmetric encoding-decoding structure and combining it with a semantic reconstruction task with a high-ratio mask, it effectively captures sentence-level semantic associations and enhances semantic representation capabilities.

[0047] (2) Efficient computing and strong cross-scenario adaptability: The lightweight single-layer decoder significantly reduces computational complexity, accelerates model convergence, reduces resource consumption, and facilitates efficient computing and deployment. The model performs well in both zero-shot and supervised learning scenarios, has strong cross-scenario adaptability, and its generalization ability is significantly better than traditional models.

[0048] (3) Convenient end-to-end application and innovation in theory and practice: The generated sentence vectors can be directly used for dense retrieval tasks, simplifying the deployment process and making them suitable for large-scale real-time retrieval systems. The organic combination of masked autoencoders and adversarial training provides a new paradigm for retrieval-oriented language model pre-training, which combines methodological innovation with practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. Those skilled in the art can derive other drawings based on these drawings without inventive effort. Among them:

[0050] Figure 1 A flowchart of a model pre-training method based on masked autoencoder and noise enhancement provided in an embodiment;

[0051] Figure 2 Schematic diagram of the encoder structure in the embodiment;

[0052] Figure 3 Schematic diagram of the decoder structure in the embodiment;

[0053] Figure 4 A schematic diagram of a dynamic noise injection architecture in an embodiment;

[0054] Figure 5 Schematic diagram of a KL divergence-guided dynamic noise injection architecture in an embodiment;

[0055] Figure 6 A block diagram of a model pre-training system based on a masked autoencoder and noise enhancement provided in an embodiment. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0057] Example 1:

[0058] In order to solve the problem of insufficient sentence-level representation capability of pre-trained models in intensive retrieval tasks in the existing technology. Figure 1As shown, an embodiment of the present invention proposes a model pre-training method based on a masked autoencoder and noise enhancement, the method comprising:

[0059] S1: Build an asymmetric encoder-decoder model, where the encoder is a full-scale deep neural network for generating an embedded representation of the input sentence, and the decoder is a lightweight single-layer neural network for reconstructing the input sentence;

[0060] S2: Apply a differentiated masking strategy to the input sentence, where the encoder mask ratio is 15%-30% and the decoder mask ratio is 50%-75% to generate differentiated training signals;

[0061] S3: A dynamic noise injection mechanism is introduced during the encoder embedding generation process. By adding noise of different amplitudes at different stages of the training process, the robustness of the asymmetric encoding-decoding model against perturbations is enhanced.

[0062] S4: Use KL divergence to guide noise generation and optimize the noise generation direction by calculating the distribution difference between the original embedding and the noisy embedding;

[0063] S5: Pre-train the asymmetric encoding-decoding model using multiple datasets and optimize the parameters of the asymmetric encoding-decoding model through a joint loss of semantic reconstruction and noise adversarial loss;

[0064] S6: The pre-trained asymmetric encoder-decoder model is directly applied to dense retrieval tasks in zero-shot or supervised learning scenarios to output a vector representation of the sentence.

[0065] As can be seen from the above technical solution, the present invention proposes a model pre-training method based on masked autoencoders and noise enhancement. First, an asymmetric encoder-decoder model is constructed. The encoder uses a full-scale neural network to convert sentences into embeddings, and the decoder uses a single-layer neural network to reconstruct sentences from masked inputs, balancing efficiency and performance. A differentiated masking strategy is then applied to the input sentences, with masking ratios ranging from 15%-30% for the encoder and 50%-75% for the decoder, to optimize embedding and reconstruction tasks. A dynamic noise injection mechanism is then introduced during the encoder embedding generation process, with the noise amplitude decreasing from high to low at the beginning to ensure retrieval stability. KL divergence is used to guide noise generation, combined with random vectors and a threshold to control the intensity, to enhance embedding discriminability. The model is then pre-trained using multiple datasets, including MS MARCO. The parameters of the asymmetric encoder-decoder model are optimized using a joint semantic reconstruction and noise adversarial loss, and performance is evaluated. Finally, the pre-trained model is applied to a dense retrieval task. This method improves the accuracy and stability of sentence representations in retrieval tasks without additional model fine-tuning, providing a new path for efficient retrieval-oriented language model pre-training.

[0066] In step S1, an asymmetric encoding-decoding model is constructed, where the encoder is a full-scale deep neural network for generating an embedded representation of the input sentence, and the decoder is a lightweight single-layer neural network for reconstructing the input sentence.

[0067] Specifically, building an asymmetric encoder-decoder model is the fundamental step of the entire pre-training method, which aims to generate text representations for input sentences suitable for dense retrieval tasks, while providing additional training supervision signals through reconstruction tasks. The encoder is designed as a neural network based on a multi-layer Transformer architecture that can capture the input sentence. (in Indicates the words, The main function of the encoder is to convert the input sentence into an embedding vector of fixed dimension. :

[0068] ;

[0069] in, is the Transformer function, are encoder parameters, , is the dimension of the encoder output embedding vector, typically 512, 768, or 1024.

[0070] Furthermore, the encoder incorporates a multi-head self-attention mechanism, a feedforward neural network, and layer normalization to ensure the stability and representational power of the deep network. Meanwhile, the decoder is designed as a lightweight single-layer neural network, also based on the Transformer architecture, but employing a single layer. Its parameter count is significantly smaller than that of the encoder (approximately 1 / 10 or less), reducing computational complexity. The decoder's task is to reconstruct the original sentence from the masked input:

[0071] ;

[0072] in, is the masked input, is a single-layer Transformer function, are decoder parameters, For the reconstruction results.

[0073] The core of this asymmetric design is that the encoder generates high-quality embedding vectors through the full-scale network to support retrieval tasks, while the decoder assists pre-training with a lightweight structure to ensure semantic consistency. First, it is converted into a token sequence through a word segmenter and special tags (such as [CLS] and [SEP]) are added. The encoder outputs a fixed-length vector, and the decoder output is consistent with the input length. To accelerate training convergence, the parameters of the encoder and decoder are and Pre-trained language model weights (such as BERT) can be used for initialization. This initialization strategy not only significantly improves model training efficiency but also achieves better performance with limited training data, laying the foundation for subsequent masking strategies and noise enhancement.

[0074] In step S2, a differentiated masking strategy is applied to the input sentence, where the encoder mask ratio is 15%-30% and the decoder mask ratio is 50%-75% to generate differentiated training signals.

[0075] Specifically, a differentiated masking strategy is applied to the input sentences. The purpose of this step is to generate differentiated training inputs for the encoder and decoder through different mask ratios, thereby enhancing the diversity of training signals and optimizing their respective task objectives.

[0076] Specifically, if Figure 2 and Figure 3 As shown, for the input sentence , the mask ratio of the encoder side is set to 15%-30%, while the mask ratio of the decoder side is set to 50%-75%, and the mask is randomly sampled, and the continuous mask is constrained not to exceed 50%. For the encoder mask operation, first determine the mask ratio (Range ), calculate the number of words that need to be masked ( represents the floor function, indicates sentence length), and then from the sentence position Randomly select positions, denoted as a set .for Each position in , the word Replace with special token [MASK] to generate masked sentences , which is of the form (like )or (like ). This mask sentence As encoder input, used to generate embedding , where Encoder(·) is the encoder function.

[0077] In contrast, the decoder masking operation uses a higher mask ratio (Range 50%-75%), calculate the number of masked words and from Randomly select positions, denoted as a set ( May contain ). Similarly, Words in Replace with , generate masked sentences ,(like )or (like ), and is used as the decoder input to reconstruct the original sentence ,in Decoder function.

[0078] The core of this differentiated masking strategy is that the encoder mask ratio is low to retain more semantic information and facilitate the generation of high-quality embeddings, while the decoder mask ratio is high to increase the reconstruction difficulty and improve the generalization ability of the model. The decoder loss function The optimization goal is:

[0079] ;

[0080] in, Indicates the first The original word at position represents the set of all masked positions in the input sentence, It is a cross entropy loss function used by the decoder to restore the masked sentence. Excessive mask ratio forces the decoder to generate high-quality vector representation.

[0081] In step S3, a dynamic noise injection mechanism is introduced in the encoder embedding generation process to enhance the robustness of the asymmetric encoding-decoding model against perturbations by adding noise of different amplitudes at different stages of the training process.

[0082] Specifically, after completing the entire autoencoder reconstruction step, a dynamic noise injection mechanism is introduced into the encoder embedding generation process, such as Figure 4 This method enhances the model's robustness against perturbations by adding dynamically adjusted noise to the embedding layer, while balancing accuracy requirements during training. This improves the stability and practicality of pre-trained models in dense retrieval tasks. This mechanism aims to ensure that the sentence embeddings generated by the encoder maintain better semantic consistency in the vector space, meeting the dual requirements of robustness and accuracy during training.

[0083] Specifically, the encoder first generates a masked sentence based on Generate raw embeddings :

[0084] ;

[0085] in, is an embedding vector of fixed dimension.

[0086] Furthermore, the task of noise injection is to embed Add random perturbations to generate noisy embeddings ,in is a noise vector that not only simulates the real-world disturbance but also dynamically adjusts its characteristics according to the training progress. In the early stages of training, this task focuses on enhancing robustness by applying a larger noise amplitude (e.g. 0.3 to 0.5 times the standard deviation) forces the model to learn robust feature representations and prevent overfitting to specific patterns; in the later stages of training, the task shifts to improving accuracy, and the noise amplitude gradually decreases to lower values ​​(such as To achieve this, the dynamic noise amplitude is defined as:

[0087] ;

[0088] in, represents the dynamic noise amplitude, represents the initial noise amplitude, Represents the attenuation rate (floating point number), used to control the speed of amplitude decrease. represents the total number of training steps, Indicates the current training step number, Indicates the minimum amplitude of noise (take ), to ensure the accuracy of the later stage.

[0089] This step uses dynamic noise injection to enhance the uncertainty and recall of the input sentence when semantically ambiguous and when encountering similar sentences. This dynamic noise injection not only enables the model to adapt to the needs of intensive retrieval under various conditions, but also maintains high-precision matching capabilities in undisturbed scenarios.

[0090] In step S4, KL divergence is used to guide noise generation, and the noise generation direction is optimized by calculating the distribution difference between the original embedding and the noisy embedding.

[0091] Specifically, KL divergence is used to guide noise generation, such as Figure 5The purpose of this step is to optimize the direction of noise generation by quantifying the distribution difference between the original embedding and the noisy embedding, so that the noise can more specifically enhance the weaknesses of the model and improve its robustness and adaptability in dense retrieval tasks. The goal of this task is not to simply add random noise, but to make the noise injection "intelligent" through information theory metrics, focusing on exposing and improving the potential defects of the encoder. Specifically, for the original embedding generated by the encoder and noisy embedding (in ), KL divergence is used to measure the difference between two embedding distributions, where and They are and By minimizing or controlling To adjust the noise vector direction, so that More inclined to deviate The noise direction is defined as a composite vector combining the KL divergence guide and randomness:

[0092] ;

[0093] in, Represents the gradient under Gaussian approximation, pointing to the direction of the largest distribution difference, represents the gradient operation on the noise vector N, represents the KL divergence, Indicates the dynamic noise amplitude; is a random vector sampled from the unit sphere, providing an exploratory perturbation; is the dynamic weight, ranging from , Indicates L2 norm normalization. To balance the contribution of KL divergence guidance and randomness, Indicates L2 norm normalization, ensuring is a unit vector. KL divergence noise can be summarized as ( Can be used in conjunction with dynamic adjustments). , the noise direction is optimized to focus on the weak areas predicted by the model, while the random vector To ensure a certain degree of exploration. In order to avoid excessive perturbation leading to embedding distortion, a threshold is set ,when Time decrease , on the contrary increase To maintain effectiveness.

[0094] Furthermore, the overall noise control direction is controlled by two parts: one is the time-dependent attenuation noise addition, and the other is the noise addition in a specific direction optimized by the KL divergence. The final noise addition method is integrated into:

[0095] ;

[0096] in, represents the dynamic noise amplitude, is the optimal noise direction under the influence of KL divergence, is the regularization coefficient, which is adjusted experimentally (if the calculated N exceeds this limit, it is scaled to the boundary), represents the original embedding The standard deviation of Represents the minimum function. The min method takes the minimum value to prevent excessive noise fluctuations from causing destructive interference to the vector representation itself, setting a fixed threshold.

[0097] In step S5, the asymmetric encoding-decoding model is pre-trained using multiple datasets, and the parameters of the asymmetric encoding-decoding model are optimized through the semantic reconstruction and noise adversarial joint loss.

[0098] Specifically, after completing the construction of the autoencoder and decoder and the noise addition part, the model was pre-trained using multiple datasets, and was fully trained on the MS MARCO (Microsoft Machine Reading Comprehension) dataset. The core task of the training is to enable the decoder to accurately restore the original sentence through semantic reconstruction. , and its loss function is defined as:

[0099] ;

[0100] in, For the mask sentence, are decoder parameters, is the sentence length, Indicates that the decoder is given a mask sentence and decoder parameters Under the condition of The word is probability.

[0101] This reconstruction task ensures that the model captures the deep semantic structure of the sentence. Meanwhile, the noise adversarial training task is performed by embedding Dynamic noise is applied (As mentioned above ), optimizes the robustness of the model to perturbations, and its loss function is defined as:

[0102] ;

[0103] in, represents a masked sentence, represents the masked sentence after adding dynamic noise, measuring the consistency of embedding, represents the square of the L2 norm, and Encoder(·) is the encoder function.

[0104] Furthermore, the overall training objective is the joint loss of semantic reconstruction and noise adversarial:

[0105] ;

[0106] in, For semantic reconstruction and noise adversarial joint loss, and are weight coefficients ranging from 0 to 1, and the encoder parameters are optimized by gradient descent. and decoder parameters In the evaluation phase, this task measures model performance through intensive retrieval tasks (such as semantic similarity matching and information retrieval), using metrics including precision, recall, and mean inverted rank (MRR) to verify the generalization ability and robustness of the model under multi-dataset training.

[0107] In step S6, the pre-trained asymmetric encoding-decoding model is directly applied to the dense retrieval task in zero-shot or supervised learning scenarios, and the retrieval task is performed through cosine similarity. The text is input into the pre-trained asymmetric encoding-decoding model to output the vector representation of the sentence, thereby achieving efficient deployment and application.

[0108] The performance of the method of the present invention on the MS_MARCO test dataset is shown in Table 1 below.

[0109] Table 1 Comparison results between the method of the present invention and the traditional method

[0110]

[0111] As can be seen from Table 1, by enhancing the noise of the decoder, the method of the present invention (KL-Noise) has shown improvements in multiple indicators compared with traditional methods in the field of text representation.

[0112] Example 2:

[0113] like Figure 6As shown, the present invention provides a model pre-training system based on masked autoencoder and noise enhancement, which is used to implement the model pre-training method based on masked autoencoder and noise enhancement in the first embodiment, specifically comprising:

[0114] An asymmetric encoding-decoding model construction module 100 is used to construct an asymmetric encoding-decoding model, where the encoder is a full-scale deep neural network for generating an embedded representation of the input sentence, and the decoder is a lightweight single-layer neural network for reconstructing the input sentence;

[0115] A differentiated masking strategy module 200 is configured to apply a differentiated masking strategy to the input sentence, wherein the encoder masking ratio is 15%-30% and the decoder masking ratio is 50%-75% to generate differentiated training signals;

[0116] Dynamic noise injection module 300, which is used to introduce a dynamic noise injection mechanism during the encoder embedding generation process, and enhance the robustness of the asymmetric encoding-decoding model against perturbations by adding noise of different amplitudes at different stages of the training process;

[0117] A KL divergence guided noise generation module 400 is configured to use KL divergence to guide noise generation and optimize the noise generation direction by calculating the distribution difference between the original embedding and the noisy embedding;

[0118] A model pre-training and parameter optimization module 500 is used to pre-train an asymmetric encoding-decoding model using multiple datasets and optimize the parameters of the asymmetric encoding-decoding model using a joint loss of semantic reconstruction and noise adversarial learning;

[0119] The model application module 600 is used to directly apply the pre-trained asymmetric encoding-decoding model to dense retrieval tasks in zero-shot or supervised learning scenarios, and output vector representations of sentences.

[0120] A model pre-training system based on a masked autoencoder and noise enhancement in this embodiment is used to implement the aforementioned model pre-training method based on a masked autoencoder and noise enhancement. Therefore, the specific implementation methods of the model pre-training system based on a masked autoencoder and noise enhancement can be seen in the embodiment part of the model pre-training method based on a masked autoencoder and noise enhancement in the previous text. For example, the asymmetric encoding-decoding model construction module 100, the differentiated mask strategy module 200, the dynamic noise injection module 300, the KL divergence guided noise generation module 400, the model pre-training and parameter optimization module 500, and the model application module 600 are respectively used to implement steps S1, S2, S3, S4, S5, and S6 in the aforementioned model pre-training method based on a masked autoencoder and noise enhancement. Therefore, its specific implementation methods can refer to the descriptions of the corresponding embodiments of each part. In order to avoid redundancy, they will not be repeated here.

[0121] Example 3:

[0122] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned model pre-training method based on masked autoencoder and noise enhancement.

[0123] Example 4:

[0124] An embodiment of the present invention provides a computer storage medium storing a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned model pre-training method based on masked autoencoder and noise enhancement.

[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0128] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications derived therefrom remain within the scope of protection of the present invention.

Claims

1. A model pre-training method based on masked autoencoder and noise enhancement, characterized in that: include: S1: Build an asymmetric encoder-decoder model, where the encoder is a full-scale deep neural network for generating an embedded representation of the input sentence, and the decoder is a lightweight single-layer neural network for reconstructing the input sentence; S2: Apply a differentiated masking strategy to the input sentence, where the encoder mask ratio is 15%-30% and the decoder mask ratio is 50%-75% to generate differentiated training signals; S3: A dynamic noise injection mechanism is introduced during the encoder embedding generation process. By adding noise of different amplitudes at different stages of the training process, the robustness of the asymmetric encoding-decoding model against perturbations is enhanced. In the dynamic noise injection mechanism, the noise amplitude is dynamically adjusted according to the training progress. Its calculation formula is: ； in, represents the dynamic noise amplitude, represents the initial noise amplitude, Indicates the attenuation rate, which is used to control the speed of amplitude decrease. represents the total number of training steps, Indicates the current training step number, Indicates the minimum amplitude of noise; S4: Use KL divergence to guide noise generation, and optimize the noise generation direction by calculating the distribution difference between the original embedding and the noisy embedding; in the KL divergence-guided noise generation, the noise direction Calculated by the following formula: ； in, Represents the gradient under Gaussian approximation, pointing to the direction of the largest distribution difference, represents the gradient operation on the noise vector N, represents the KL divergence, Indicates the dynamic noise amplitude; is used to measure the difference between two embedding distributions, and The original embedding and noisy embedding The probability distribution of is a random vector sampled from the unit sphere, providing an exploratory perturbation; is the dynamic weight, ranging from , Indicates L2 norm normalization; S5: Pre-train the asymmetric encoding-decoding model using multiple datasets and optimize the parameters of the asymmetric encoding-decoding model through a joint loss of semantic reconstruction and noise adversarial loss; S6: The pre-trained asymmetric encoder-decoder model is directly applied to dense retrieval tasks in zero-shot or supervised learning scenarios to output a vector representation of the sentence.

2. The model pre-training method based on masked autoencoder and noise enhancement according to claim 1, characterized in that In the asymmetric encoding-decoding model, the encoder adopts a multi-layer Transformer architecture to output a fixed-dimensional embedding vector, and the decoder adopts a single-layer Transformer architecture. Its parameter number does not exceed 1 / 10 of the encoder's parameter number, and the parameters of the encoder and decoder are initialized using a pre-trained language model.

3. The model pre-training method based on masked autoencoder and noise enhancement according to claim 1, characterized in that Step S2 specifically includes: For the encoder mask operation, first determine the mask ratio , the range is , calculate the number of words that need to be masked , then from the sentence position Randomly select positions, denoted as a set ;for Each position in , the word Replace with special token [MASK] to generate masked sentences , which is of the form , or , ; This mask sentence As encoder input, used to generate embedding and the masked word vectors ; In contrast, the decoder mask operation uses a higher mask ratio , ranging from 50% to 75%, calculate the number of masked words and from Randomly select positions, denoted as a set ; Similarly, Words in Replace with special mark , generate masked sentences , , or , , and as decoder input and position encoding Together they are used to reconstruct the original sentence ; in, represents the floor function, represents the length of the sentence, Encoder(·) is the encoder function, Decoder function.

4. The model pre-training method based on masked autoencoder and noise enhancement according to claim 1, characterized in that The noisy embedding Expressed as: ； in, represents the dynamic noise amplitude, is the optimal noise direction under the influence of KL divergence, is the regularization coefficient, which is adjusted through experiments. represents the original embedding The standard deviation of represents the minimum function.

5. The model pre-training method based on masked autoencoder and noise enhancement according to claim 1, characterized in that The semantic reconstruction and noise adversarial joint loss is: ； in, For semantic reconstruction and noise adversarial joint loss, and are weight coefficients ranging from 0 to 1. is the semantic reconstruction loss, for noise combating losses.

6. The model pre-training method based on masked autoencoder and noise enhancement according to claim 5, characterized in that The semantic reconstruction loss is: ； in, For the mask sentence, are decoder parameters, is the sentence length, Indicates that the decoder is given a mask sentence and decoder parameters Under the condition of The word is probability.

7. The model pre-training method based on masked autoencoder and noise enhancement according to claim 5, characterized in that The noise resistance loss is: ； in, represents a masked sentence, represents the masked sentence after adding dynamic noise, represents the square of the L2 norm, and Encoder(·) is the encoder function.

8. A model pre-training system based on masked autoencoder and noise enhancement, characterized in that: The system is used to implement the model pre-training method based on masked autoencoder and noise enhancement according to any one of claims 1 to 7, specifically comprising: Asymmetric encoding-decoding model building module, used to build an asymmetric encoding-decoding model, where the encoder is a full-scale deep neural network for generating embedded representations of input sentences, and the decoder is a lightweight single-layer neural network for reconstructing the input sentences; Differentiated masking strategy module, which applies a differentiated masking strategy to the input sentence, with an encoder masking ratio of 15%-30% and a decoder masking ratio of 50%-75% to generate differentiated training signals; Dynamic noise injection module, which is used to introduce a dynamic noise injection mechanism during the encoder embedding generation process. By adding noise of different amplitudes at different stages of the training process, it enhances the robustness of the asymmetric encoding-decoding model against perturbations. The KL divergence-guided noise generation module is used to use KL divergence to guide noise generation and optimize the noise generation direction by calculating the distribution difference between the original embedding and the noisy embedding; Model pre-training and parameter optimization module, which is used to pre-train the asymmetric encoding-decoding model using multiple datasets and optimize the parameters of the asymmetric encoding-decoding model through semantic reconstruction and noise adversarial joint loss; The model application module is used to directly apply the pre-trained asymmetric encoding-decoding model to dense retrieval tasks in zero-shot or supervised learning scenarios, and output the vector representation of the sentence.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the model pre-training method based on masked autoencoder and noise enhancement as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that The computer storage medium stores a computer software product, which includes several instructions for enabling a computer device to execute the model pre-training method based on masked autoencoder and noise enhancement as described in any one of claims 1 to 7.

Images