Processing method and device for retrieving enhanced language model based on variational auto-encoder architecture

Abstract

The embodiment of the invention relates to a processing method and device for retrieving an enhanced language model based on a variational auto-encoder architecture. The method comprises the following steps: introducing a low-rank enhancer for a basic model to obtain a basic revision model; performing data acquisition on the public question and answer data set to obtain a first data set training basic revision model; constructing a second data set according to the target text library after training is finished, and combining the first data set and the second data set to obtain a third data set; constructing a feature vector library of the target text library; a retriever and a feature aggregation module are introduced into the basic revision model to form a retrieval enhancement language model, and the retrieval enhancement language model is in butt joint with the feature vector library; training a retrieval enhancement language model based on the third data set; and after the training is finished, processing a natural language question and answer task by using a retrieval enhanced language model. The quality of the generated text can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a processing method and apparatus for a retrieval-enhanced language model based on a variational autoencoder architecture. Background Technology

[0002] With the rapid development of Natural Language Processing (NLP) technology, generative large language models (LLMs) based on the Variational Auto-Encoder (VAE) architecture have shown potential in text generation tasks.

[0003] However, current generative LLMs based on VAE architecture still have some shortcomings when applied to complex question-answering scenarios: 1) The posterior collapse problem is prominent. During training, the encoder of the VAE architecture tends to ignore input information, which makes the latent variables unable to effectively carry the query semantics, causing the decoder to degenerate into an unconditional language model, and reducing the relevance between the generated content and the input question; 2) The external knowledge fusion mechanism is lacking. Most generative LLMs based on VAE architecture do not have a retrieval mechanism, which cannot dynamically retrieve and utilize the latest information in the external knowledge base during inference, resulting in poor timeliness of the generated answers; 3) A few models introduce a retrieval mechanism to correct the defect 2), but their retrieval enhancement method is to directly concatenate the query and the retrieved text. This simple concatenation method is limited by the model context length and inputs the retrieved information as noise in the original text, failing to achieve deep fusion at the latent semantic level, resulting in factual bias in the generated content.

[0004] To address the aforementioned shortcomings, we propose an improved solution: a retrieval unit, a feature aggregation module, and a low-rank enhancer are introduced into the traditional generative LLM architecture of VAE. The retrieval unit performs external knowledge retrieval based on feature retrieval methods, thereby improving the quality and timeliness of the generated text. The feature aggregation module performs feature fusion on the query and retrieved text features using an implicit aggregation method with a Gaussian mixture distribution, thereby reducing feature noise, increasing feature richness, and improving generation quality and factual accuracy. The low-rank enhancer forces the use of latent variables at each layer of the decoder, thereby preventing posterior collapse and improving generation quality. How to implement this improved solution is the technical problem that this invention aims to solve. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a processing method, apparatus, electronic device, and computer-readable storage medium for a retrieval-augmented language model based on a variational autoencoder architecture. This invention first uses any generative large language model implemented based on a VAE architecture, which has already undergone natural language pre-training and question-answering task-oriented fine-tuning, as the base model. This base model includes an encoder, a latent variable module, and a decoder. A low-rank enhancer is introduced into the base model to obtain a corresponding modified base model. Then, a first dataset is constructed by collecting publicly available question-answering datasets; and the modified model is trained based on the first dataset. After the modified model training is completed, question-answer text pairs are assembled from an external knowledge base (i.e., the target text library), and a second dataset is constructed based on the assembly results. The first and second datasets are then merged to obtain a third dataset; and a feature vector library corresponding to the target text library is constructed using the modified base model. Then, a retrieval unit and a feature aggregation module are introduced into the modified model to form retrieval-augmented language models (RALMs), which are then interfaced with the feature vector library; and the retrieval-augmented language model is trained using the third dataset. After the augmentation model is trained, the final retrieval-enhanced language model is used to process natural language question-answering tasks. This invention, based on a VAE architecture, implements a retrieval-enhanced language model that improves the quality and timeliness of generated text through a retrieval engine, reduces feature noise, increases feature richness, and enhances generation quality and factual accuracy through a feature aggregation module, and prevents posterior collapse and improves generation quality through a low-rank enhancer.

[0006] To achieve the above objectives, a first aspect of the present invention provides a processing method for a retrieval enhancement language model based on a variational autoencoder architecture, the method comprising: Any generative large language model based on a variational autoencoder architecture, which has completed natural language pre-training and question-answering task-oriented fine-tuning, is used as the base model; the base model is used to encode the features of the query text X input to the model, generate corresponding latent variables based on the encoded features, and decode the generated text according to the latent variables to obtain the corresponding generated text Y and output it; the base model includes an encoder, a latent variable module, and a decoder. A corresponding low-rank enhancer is introduced into the decoder of the base model to obtain a corresponding base modified model; the base modified model includes the encoder, the latent variable module, the decoder and the low-rank enhancer; the low-rank enhancer is used to enhance the latent variable features of each layer output of the decoder through low-rank transformation; The first dataset was obtained by performing big data collection on the publicly available question-and-answer dataset. The modified model is trained based on the first dataset. After the modified model is trained, question-answer text pairs are assembled according to a preset target text library, and a second dataset is constructed based on the assembly results; then the first and second datasets are merged to obtain the corresponding third dataset; the target text library includes multiple target texts; Based on the aforementioned modified model, the target text library is feature-encoded to obtain a corresponding feature vector library. A retrieval unit and a feature aggregation module are introduced into the modified model to form a retrieval-enhanced language model. This retrieval-enhanced language model is then integrated with the feature vector library. The feature vector library includes multiple text feature vectors, each corresponding one-to-one with the target text. The retrieval-enhanced language model includes the encoder, the latent variable module, the retrieval unit, the feature aggregation module, the decoder, and the low-rank enhancer. The retrieval unit is integrated with the feature vector library and is used to perform similar feature retrieval on the feature vector library based on the output features of the latent variable module. The feature aggregation module is used to aggregate the output features of the latent variable module and the retrieval unit and input the aggregated features into the decoder for decoding. The retrieval enhancement language model is trained based on the third dataset. After the augmented model is trained, the retrieval augmented language model is used to process natural language question answering tasks.

[0007] Preferably, the first, second, and third datasets each consist of multiple question-and-answer text records; the question-and-answer text records include question text and answer text. The publicly available question-and-answer datasets include at least the WebQA dataset, the DuReader dataset, and the LCQMC dataset.

[0008] Preferably, in the basic model, the input terminal of the encoder is connected to the input terminal of the latent variable module; the output terminal of the latent variable module is connected to the input terminal of the decoder. The encoder is implemented based on the encoding model of the Transformer framework; the encoder is used to perform Gaussian distribution feature encoding on the query text X to obtain the corresponding encoded feature vector E and send it to the latent variable module; Wherein, the encoded feature vector E is composed of the mean feature vector µ X Log-variance vector composition; σ is the variance feature vector of the query text X; X Let X be the standard deviation feature vector of the query text X; The latent variable module is used to construct the corresponding latent variable Z based on the encoded feature vector E. X Send to the decoder; Wherein, the latent variable Z X The construction method is as follows: ; ε is the Gaussian noise tensor; The decoder is implemented based on the Transformer framework decoding model; the decoding network of the decoder consists of L sequentially connected layer l decoding modules, where 1 ≤ layer index l ≤ L, and L is the total number of decoding modules in the decoder; the input and output features of the layer l decoding module are denoted as the corresponding feature vector H. l-1 H l ; In the base model, the decoder is used to decode the model in an autoregressive manner according to the Transformer framework, and the decoding network is used to decode the latent variable Z. X Perform feature decoding and text generation processing based on the decoded feature vectors to obtain the corresponding generated text Y output; In the basic model, when the decoder uses the decoding network for decoding: it converts the latent variable Z... X The input feature vector H0 of the first-layer decoding module is fed into the decoding network for layer-by-layer decoding, and the feature vector H output by the Lth-layer decoding module is... L This is the decoded feature vector output by the decoding network.

[0009] Preferably, in the basic modified model, the output of the encoder is connected to the input of the latent variable module; the output of the latent variable module is connected to the inputs of the decoder and the low-rank enhancer respectively; the decoder is also interconnected with the low-rank enhancer. The encoder is used to perform Gaussian distribution feature encoding on the query text X to obtain the corresponding encoded feature vector E, which is then sent to the latent variable module. The latent variable module is used to construct the corresponding latent variable Z based on the encoded feature vector E. X Send to the decoder and the low-rank enhancer; In the aforementioned modified model, the decoder is used to decode the model's autoregressive decoding method according to the Transformer framework, and the decoding network is used to decode the latent variable Z. X Feature decoding is performed, and during the decoding process, the low-rank enhancer is used to enhance the latent variable features of the decoded feature vector. Based on the enhanced decoded feature vector, text generation processing is performed to obtain the corresponding generated text Y output. In the aforementioned basic modified model, when the decoder uses the decoding network for decoding: it converts the latent variable Z... XThe input feature vector H0 of the first-layer decoding module is decoded to obtain the corresponding feature vector H1. This feature vector H1 is then input into the corresponding first-layer low-rank module in the low-rank enhancer for latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V1. This enhanced feature vector V1 is then used as the new feature vector H1 and input into the second-layer decoding module, and so on. Finally, the feature vector H0 obtained from the Lth-layer decoding module is... L The corresponding Lth-level low-rank module in the low-rank enhancer is input to perform latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V. L and the enhanced feature vector V L As the new feature vector H L and the new feature vector H L As the decoded feature vector for feature enhancement; The low-rank enhancer includes L low-rank modules of layer l; each of the low-rank modules of the low-rank enhancer is used to perform low-rank decomposition based on the latent variable Z. X The feature vector H corresponding to the l-th layer decoding module l The latent variable features are enhanced to obtain the corresponding enhanced feature vector V. l ; Wherein, the enhanced feature vector V l The calculation method is as follows: , R is the preset total number of ranks, where 1 ≤ rank index r ≤ R; The content path weight matrix of the l-th low-rank module is... Let r be the potential path weight matrix of the l-th low-rank module; , The r-th low-rank weight matrix component that makes up the l-th low-rank module; ⊙ is the Hadamard product.

[0010] Preferably, in the retrieval-enhanced language model, the output of the encoder is connected to the input of the latent variable module; the output of the latent variable module is connected to the inputs of the retrieval unit and the feature aggregation module; the output of the retrieval unit is connected to the input of the feature aggregation module; the retrieval unit is also interconnected with the feature vector library; the output of the feature aggregation module is connected to the inputs of the decoder and the low-rank enhancer; the decoder is also interconnected with the low-rank enhancer. The encoder is used to perform Gaussian distribution feature encoding on the query text X to obtain the corresponding encoded feature vector E, which is then sent to the latent variable module. The latent variable module is used to construct the corresponding latent variable Z based on the encoded feature vector E.X Send to the retriever and the feature aggregation module; The retrieval unit is used to perform a cosine similarity algorithm on the latent variable Z. X The vector similarity with each of the text feature vectors in the feature vector library is calculated; and the K text feature vectors corresponding to the top K similarity scores in descending order of similarity are denoted as the corresponding latent variables Z. k ; and each of the aforementioned latent variables Z k The corresponding vector similarity is denoted as similarity d. k ; and by each of the aforementioned latent variables Z k and the corresponding similarity d k Form a corresponding search record s k (Z k ,d k ); and from the obtained K retrieved records s k (Z k ,d k ) form the corresponding search record sequence {s k (Z k ,d k Send the result to the feature aggregation module; the total number of searches, K, is a preset positive integer; The feature aggregation module is used to perform the retrieval record sequence {s} k (Z k ,d k )} and the latent variable Z X The corresponding latent variable Z is obtained by performing feature aggregation and sent to the decoder and the low-rank enhancer; The feature aggregation method for the latent variable Z is as follows: , , , ; τ is the preset temperature coefficient; w k For the latent variable Z k The corresponding weighting coefficient; Z S The corresponding retrieval feature vector; concatenate is the vector concatenation function; Z con The corresponding feature concatenation vector, Z con The vector length is the retrieved feature vector Z. S With the latent variable Z X The sum of vector lengths; W Z B ZThe weight matrix and bias vector corresponding to the feature aggregation module; the vector shape of the latent variable Z and the latent variable Z X Maintain consistency; In the retrieval enhancement language model, the decoder is used to perform autoregressive decoding of the decoding model according to the Transformer framework. The decoding network is used to perform feature decoding on the latent variable Z. During the decoding process, the low-rank enhancer is used to enhance the latent variable features of the decoded feature vector. Based on the feature-enhanced decoded feature vector, text generation processing is performed to obtain the corresponding generated text Y output. In the retrieval enhancement language model, when the decoder uses the decoding network for decoding: the latent variable Z is used as the input feature vector H0 of the first-layer decoding module to decode and obtain the corresponding feature vector H1; the feature vector H1 is then input into the corresponding first-layer low-rank module in the low-rank enhancer for latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V1; the enhanced feature vector V1 is then used as the new feature vector H1 and input into the second-layer decoding module, and so on, until the feature vector H0 obtained by the Lth-layer decoding module is finally decoded. L The corresponding Lth-level low-rank module in the low-rank enhancer is input to perform latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V. L and the enhanced feature vector V L As the new feature vector H L and the new feature vector H L As the decoded feature vector for feature enhancement; In the retrieval enhancement language model, each of the l-th layer low-rank modules of the low-rank enhancer is used to decompose the latent variable Z into the feature vector H corresponding to the l-th layer decoding module according to the low-rank decomposition method. l The latent variable features are enhanced to obtain the corresponding enhanced feature vector V. l ; Wherein, the enhanced feature vector V l The calculation method is as follows: .

[0011] Preferably, the step of training the modified model based on the first dataset specifically includes: Step 61: Divide the first dataset into two sub-datasets according to a preset first segmentation ratio, denoted as the first training set and the first evaluation set. Wherein, both the first training set and the first evaluation set consist of multiple question-and-answer text records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; Step 62: Take the first question-and-answer text record of the first training set as the current training record; Step 63: Input the question text from the current training record as the corresponding query text X into the base modified model for processing; save the process data of this processing; and record the answer text from the current training record as the corresponding expected text Y. * ; and for the desired text Y * Perform word segmentation sequence conversion to obtain the corresponding word segmentation sequence { }; Where 1 ≤ word segmentation index i ≤ N 1 N 1 The desired text Y is as described above. * The corresponding total number of word segments; The desired text Y is as described above. * The i-th segmented text; Step 64: Input the process data of this model processing into the preset first model loss function L. M1 The corresponding first loss value is obtained through calculation; Wherein, the first model loss function L M1 Based on the first reconstruction loss function L RECON1 and the first divergence loss function L KL1 Implementation, specifically: , , ; β1 is the balancing weight parameter; This is the preset start marker; Given the latent variable Z X and word segmentation sequence Under the given conditions, the i-th segmented text generated by the model is the expected text Y. * The i-th segmented text The probability; θ1 is the set of model parameters of the decoder and the low-rank enhancer of the basic modified model; D KL () represents the Kullback-Leibler divergence function; To be related to the latent variable Z X The corresponding prior distribution is a pre-defined standard Gaussian distribution; To be related to the latent variable Z X The corresponding posterior distribution; θ2 is the model parameter of the encoder in the basic modified model; Step 65: Identify whether the first loss value meets a preset first loss value range; if the first loss value meets the first loss value range, identify whether the current training record is the last question-and-answer text record in the first training set; if yes, proceed to step 66; otherwise, take the next question-and-answer text record in the first training set as the new current training record and return to step 63; if the first loss value does not meet the first loss value range, optimize the first model based on a preset first model optimizer to make the first model loss function L... M1 The model parameters of the basic modified model are modulated once in the direction that reaches the minimum value, and the process returns to step 63 when the modulation ends. The first model optimizer includes the Adam optimizer and the SGD optimizer. Step 66, record the total number of records in the first evaluation set as... The first evaluation set is iterated through all the question-and-answer text records in the first evaluation set. During this iteration, the currently iterated question-and-answer text record is used as the current evaluation record. The question text of the current evaluation record is used as the corresponding query text X and input into the basic modified model for processing to obtain the corresponding generated text Y. The generated text Y and the answer text of the current evaluation record are input into the basic modified model for processing, and the two latent variables Z obtained are... X The predicted text features and label text features are used as the corresponding text features; and the similarity d is calculated by using the cosine similarity algorithm to obtain the similarity between the predicted and label text features corresponding to the current evaluation record. j ; and at the end of this round of traversal, the obtained The similarity d j Substitute into the preset first model evaluation function L EV1 The corresponding first evaluation value is obtained through calculation; Where 1 ≤ similarity index j ≤ ; The first model evaluation function L EV1 Implemented based on the RMSE function, specifically as follows: ; Step 67: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 61; if it does, stop training and confirm that the training of the modified model of the basic modified model has ended.

[0012] Preferably, the step of assembling question-and-answer text pairs according to a preset target text library and constructing a second dataset based on the assembly results specifically includes: Each target text in the target text library is used as the current answer text; the current answer text is input into a preset question conversion processing interface for processing, and the output text of the question conversion processing interface is used as the corresponding current question text; the current question text and the current answer text form a corresponding question and answer text record; and all the question and answer text records corresponding to the target text library form the corresponding second dataset; The question conversion processing interface is used to provide a corresponding question for the current answer text input to the interface and output it.

[0013] Preferably, the step of performing feature encoding on the target text library based on the basic modified model to obtain the corresponding feature vector library specifically includes: Each target text in the target text library is taken as the current text; and the current text is used as the corresponding query text X and input into the basic redesign model for processing, and the latent variable Z obtained from this processing is used as the current text. X As the corresponding text feature vector; and the corresponding feature vector library is composed of all the obtained text feature vectors.

[0014] Preferably, the step of training the retrieval enhancement language model based on the third dataset specifically includes: Step 91: Divide the third dataset into two sub-datasets according to the preset second segmentation ratio, and denoted as the corresponding second training set and second evaluation set; The second training set and the second evaluation set each consist of multiple question-and-answer text records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio. Step 92: Take the first question-and-answer text record of the second training set as the current training record; Step 93: Input the question text from the current training record as the corresponding query text X into the retrieval enhancement language model for processing; save the process data of this processing; and record the answer text from the current training record as the corresponding expected text Y. * ; and for the desired text Y * Perform word segmentation sequence conversion to obtain the corresponding word segmentation sequence { }; Where 1 ≤ word segmentation index u ≤ N 2 N 2 The desired text Y is as described above. * The corresponding total number of word segments; The desired text Y is as described above. * The uth segmented text; Step 94: Input the process data of this model processing into the preset second model loss function L. M2 The corresponding second loss value is obtained through calculation; Wherein, the second model loss function L M2 Based on the second reconstruction loss function L RECON2 Second divergence loss function L KL2 Implementation, specifically: , , ; β2 is the balancing weight parameter; This is the preset start marker; Given the latent variable Z and the word segmentation sequence Under the given conditions, the u-th segmented text generated by the model is the expected text Y. * The uth segmented text The probability of θ3; θ3 is the set of model parameters of the decoder and the low-rank enhancer of the retrieval enhancement language model; D KL () represents the Kullback-Leibler divergence function; The prior distribution corresponding to the latent variable Z is specifically a pre-defined standard Gaussian distribution; θ4 represents the posterior distribution corresponding to the latent variable Z; θ4 is the set of model parameters for the encoder and the feature aggregation module of the retrieval enhancement language model. Step 95: Identify whether the second loss value meets the preset second loss value range; if the second loss value meets the second loss value range, identify whether the current training record is the last question-and-answer text record of the second training set; if yes, proceed to step 96; otherwise, take the next question-and-answer text record of the second training set as the new current training record and return to step 93; if the second loss value does not meet the second loss value range, based on the preset second model optimizer, move towards making the second model loss function L... M2 The direction that reaches the minimum value is used to modulate the model parameter set of the feature aggregation module, the decoder and the low-rank enhancer of the retrieval enhancement language model in one round, and the process returns to step 93 when the modulation in this round ends. The second model optimizer includes the Adam optimizer and the SGD optimizer; Step 96, record the total number of records in the second evaluation set as... The second evaluation set is iterated through all the question-and-answer text records in the second evaluation set. During this iteration, the currently iterated question-and-answer text record is used as the current evaluation record. The question text of the current evaluation record is used as the corresponding query text X and input into the retrieval enhancement language model for processing to obtain the corresponding generated text Y. The generated text Y and the answer text of the current evaluation record are input into the retrieval enhancement language model for processing, and the two latent variables Z obtained are processed. X The predicted text features and label text features are used as the corresponding text features; and the similarity d is calculated by using the cosine similarity algorithm to obtain the similarity between the predicted and label text features corresponding to the current evaluation record. v ; and at the end of this round of traversal, the obtained The similarity d v Substitute into the preset second model evaluation function L EV2 The corresponding second evaluation value is obtained through calculation; Where 1 ≤ similarity index v ≤ ; The second model evaluation function L EV2 Implemented based on the RMSE function, specifically as follows: ; Step 97: Identify whether the second evaluation value meets the preset second evaluation value range; if not, return to step 91; if yes, stop training and confirm that the training of the enhanced language model for retrieval has ended.

[0015] A second aspect of the present invention provides an apparatus for implementing the processing method of the retrieval enhancement language model based on the variational autoencoder architecture described in the first aspect above. The apparatus includes: a base model selection module, a modified model setting module, a first data acquisition module, a modified model training module, a second data acquisition module, a final model setting module, a final model training module, and a model application module. The base model selection module is used to select any generative large language model based on a variational autoencoder architecture that has completed natural language pre-training and question-answering task-oriented fine-tuning as the base model; the base model is used to encode the features of the query text X input to the model, generate corresponding latent variables based on the encoded features, and decode the generated text according to the latent variables to obtain the corresponding generated text Y and output it; the base model includes an encoder, a latent variable module and a decoder. The modified model setting module is used to introduce a corresponding low-rank enhancer to the decoder of the base model to obtain a corresponding base modified model; the base modified model includes the encoder, the latent variable module, the decoder and the low-rank enhancer; the low-rank enhancer is used to enhance the latent variable features of each layer output of the decoder through low-rank transformation; The first data acquisition module is used to perform big data collection on the public question and answer dataset to obtain the corresponding first dataset; The modified model training module trains the modified model based on the first dataset on the basic modified model; The second data acquisition module is used to assemble question-and-answer text pairs according to a preset target text library after the modified model training is completed, and to construct a second dataset based on the assembly results; and to merge the first and second datasets to obtain the corresponding third dataset; the target text library includes multiple target texts; The final model setting module performs feature encoding on the target text library based on the basic modified model to obtain a corresponding feature vector library; and introduces a retrieval unit and a feature aggregation module into the basic modified model to form a retrieval-enhanced language model; and interfaces the retrieval-enhanced language model with the feature vector library; the feature vector library includes multiple text feature vectors, each corresponding one-to-one with the target text; the retrieval-enhanced language model includes the encoder, the latent variable module, the retrieval unit, the feature aggregation module, the decoder, and the low-rank enhancer; the retrieval unit interfaces with the feature vector library and is used to perform similar feature retrieval on the feature vector library based on the output features of the latent variable module; the feature aggregation module is used to aggregate features from the output features of the latent variable module and the retrieval unit and input the aggregated features into the decoder for decoding; The final model training module trains the retrieval enhancement language model based on the third dataset. The model application module is used to process natural language question answering tasks using the retrieval-enhanced language model after the enhanced model training is completed.

[0016] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver; The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0018] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for processing a retrieval-enhanced language model based on a variational autoencoder architecture. As described above, this invention first uses any generative large language model implemented based on a VAE architecture, which has completed natural language pre-training and question-answering task-oriented fine-tuning, as a base model. This base model includes an encoder, a latent variable module, and a decoder. A low-rank enhancer is introduced into the base model to obtain a corresponding modified base model. Then, a first dataset is constructed by collecting publicly available question-answering datasets; and the modified model is trained based on the first dataset. After the modified model training is completed, question-answer text pairs are assembled from an external knowledge base (i.e., the target text library), and a second dataset is constructed based on the assembly results. The first and second datasets are merged to obtain a third dataset; and a feature vector library corresponding to the target text library is constructed using the modified base model. Then, a retrieval unit and a feature aggregation module are introduced into the modified model to form a retrieval-enhanced language model, which is then connected to the feature vector library; and the retrieval-enhanced language model is trained based on the third dataset. After the enhanced model training is completed, the final retrieval-enhanced language model is used to process natural language question-answering tasks. The retrieval enhancement language model implemented based on the VAE architecture of this invention improves the quality and timeliness of generated text through the retrieval engine, reduces feature noise, improves feature richness, and enhances generation quality and factual accuracy through the feature aggregation module, and prevents posterior collapse and improves generation quality through the low-rank enhancer. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a retrieval enhancement language model processing method based on a variational autoencoder architecture provided in Embodiment 1 of the present invention; Figure 2 A schematic diagram of the modules of the basic model provided in Embodiment 1 of the present invention; Figure 3 A schematic diagram of the modules of the basic modified model provided in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the modules of the retrieval enhancement language model provided in Embodiment 1 of the present invention; Figure 5 This is a module structure diagram of a processing device for a retrieval enhancement language model based on a variational autoencoder architecture, provided in Embodiment 2 of the present invention. Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] Embodiment 1 of the present invention provides a processing method for a retrieval enhancement language model based on a variational autoencoder architecture, such as... Figure 1 The schematic diagram shows a retrieval enhancement language model processing method based on a variational autoencoder architecture provided in Embodiment 1 of the present invention. The method mainly includes the following steps: Step 1: Use any generative large language model based on a variational autoencoder architecture that has completed natural language pre-training and question-answering task-oriented fine-tuning as the base model.

[0022] Here, the basic model in this embodiment of the invention is used to encode the query text X input to the model, generate corresponding latent variables based on the encoded features, decode the generated text according to the latent variables to obtain the corresponding generated text Y, and output it; for example Figure 2 As shown in the schematic diagram of the basic model provided in Embodiment 1 of the present invention, the basic model includes an encoder, a latent variable module, and a decoder.

[0023] In the basic model, the input of the encoder is connected to the input of the latent variable module; the output of the latent variable module is connected to the input of the decoder.

[0024] The encoder in this embodiment of the invention is implemented based on the encoding model of the Transformer framework. The encoder in this embodiment is used to perform Gaussian distribution feature encoding on the query text X to obtain the corresponding encoded feature vector E, which is then sent to the latent variable module.

[0025] Here, the encoded feature vector E in this embodiment of the invention is composed of the mean feature vector µ. X Log-variance vector composition; σ is the variance eigenvector of the query text X; X The standard deviation feature vector of the query text X.

[0026] The latent variable module in this embodiment of the invention is used to construct the corresponding latent variable Z based on the encoded feature vector E. X Send to the decoder.

[0027] Here, the potential variable Z in the embodiments of the present invention X The construction method is as follows: ; Where ε is the Gaussian noise tensor.

[0028] The decoder in this embodiment of the invention is implemented based on the decoding model of the Transformer framework. The decoding network of the decoder in this embodiment consists of L sequentially connected layer l decoding modules, where 1 ≤ layer index l ≤ L, and L is the total number of decoding modules in the decoder. The input and output features of the layer l decoding module are denoted as the corresponding feature vector H. l-1 H l .

[0029] In the base model, the decoder is used to decode the model's autoregressive decoding method according to the Transformer framework, using the decoding network to decode the latent variable Z. X Perform feature decoding and text generation processing based on the decoded feature vectors to obtain the corresponding generated text Y output.

[0030] In the basic model, when the decoder uses the decoding network for decoding: the latent variable Z... X The input feature vector H0 of the first-layer decoding module is fed into the decoding network for layer-by-layer decoding, and the feature vector H output by the Lth-layer decoding module is... L As the decoded feature vector output by the decoding network.

[0031] Step 2: Introduce the corresponding low-rank enhancer into the decoder of the basic model to obtain the corresponding modified basic model.

[0032] Here, the model input and output of the basic modified model in this embodiment of the invention are consistent with the basic model. For example... Figure 3 As shown in the schematic diagram of the basic modified model provided in Embodiment 1 of the present invention, the basic modified model of the present invention includes an encoder, a latent variable module, a decoder, and a low-rank enhancer; wherein, the low-rank enhancer is used to enhance the latent variable features of each layer output of the decoder through low-rank transformation.

[0033] In the basic modified model, the output of the encoder is connected to the input of the latent variable module; the output of the latent variable module is connected to the input of the decoder and the low-rank enhancer respectively; the decoder is also interconnected with the low-rank enhancer.

[0034] The encoder is used to encode the query text X using a Gaussian distribution to obtain the corresponding encoded feature vector E, which is then sent to the latent variable module; this is similar to the encoder in the basic model.

[0035] The latent variable module is used to construct the corresponding latent variable Z based on the encoded feature vector E. X Send to the decoder and low-rank enhancer; this is similar to the latent variable module of the base model.

[0036] In the basic modified model, the decoder is used to decode the model's autoregressive decoding method according to the Transformer framework, and the decoding network is used to decode the latent variable Z. X Feature decoding is performed, and during the decoding process, the decoded feature vector is enhanced with latent variable features through a low-rank enhancer. Based on the enhanced decoded feature vector, text generation processing is performed to obtain the corresponding generated text Y output.

[0037] In the basic modified model, when the decoder uses the decoding network for decoding: the latent variable Z... X The input feature vector H0 of the first-layer decoding module is decoded to obtain the corresponding feature vector H1. Feature vector H1 is then input into the corresponding first-layer low-rank module in the low-rank enhancer for latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V1. Enhanced feature vector V1 is then used as the new feature vector H1 and input into the second-layer decoding module, and so on. Finally, the feature vector H0 obtained from the Lth-layer decoding module is... L The corresponding Lth-level low-rank module in the input low-rank enhancer is used to perform latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V. L And will strengthen the feature vector V L As the new feature vector H L and the new feature vector H L Decoding feature vectors as feature enhancement.

[0038] The low-rank enhancer in this embodiment of the invention includes L low-rank modules of layer l; the low-rank modules of the low-rank enhancer correspond one-to-one with the low-rank modules of the decoder of layer l, such as... Figure 3 As shown. In this embodiment of the invention, each of the l-th layer low-rank modules of the low-rank enhancer is used to perform low-rank decomposition based on the latent variable Z. X The feature vector H corresponding to the l-th layer decoding module l The latent variable features are enhanced to obtain the corresponding enhanced feature vector V. l .

[0039] Here, the enhanced feature vector V in this embodiment of the invention l The calculation method is as follows: ; Where R is the preset total number of ranks, 1 ≤ rank index r ≤ R; For the r-th content path weight matrix of the l-th low-rank module, Let r be the potential path weight matrix of the l-th low-rank module; , The r-th low-rank weight matrix component that makes up the l-th low-rank module; ⊙ is the Hadamard product.

[0040] Step 3: Collect big data from the public question-and-answer dataset to obtain the corresponding first dataset.

[0041] Here, the first dataset in this embodiment of the invention consists of multiple question-and-answer text records; the question-and-answer text records include question text and answer text. The publicly available question-and-answer datasets in this embodiment of the invention include at least the WebQA dataset, the DuReader dataset, and the LCQMC dataset.

[0042] Step 4: Train the modified model based on the first dataset. Specifically, it includes: Step 41, dividing the first dataset into two sub-datasets according to a preset first segmentation ratio, denoted as the corresponding first training set and first evaluation set; Here, the first segmentation ratio in this embodiment of the invention is a preset ratio parameter, such as 8:2; both the first training set and the first evaluation set consist of multiple question-and-answer text records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; Step 42: Take the first question-and-answer text record of the first training set as the current training record; Step 43: Input the question text of the current training record as the corresponding query text X into the basic modified model for processing; save the process data of this processing; and record the answer text of the current training record as the corresponding expected text Y. * ; and for the expected text Y * Perform word segmentation sequence conversion to obtain the corresponding word segmentation sequence { }; Here, 1 ≤ word segmentation index i ≤ N 1 N 1 For the current expected text Y * The corresponding total number of word segments; For the current expected text Y * The i-th segmented text; Step 44: Input the process data of this model processing into the preset first model loss function L. M1 The corresponding first loss value is obtained through calculation; Here, the first model loss function L in this embodiment of the invention M1 Based on the first reconstruction loss function L RECON1 and the first divergence loss function L KL1 Implementation, specifically: , , ; Where β1 is the balancing weight parameter; This is the preset start marker; For a given latent variable Z X and word segmentation sequence Under the given conditions, the i-th segmented text generated by the model is the expected text Y. * The i-th segmented text The probability of θ1; the set of model parameters for the decoder and low-rank enhancer of the modified model based on θ1; D KL () represents the Kullback-Leibler divergence function; To be related to the latent variable Z X The corresponding prior distribution is a pre-defined standard Gaussian distribution; To be related to the latent variable Z X The corresponding posterior distribution; θ2 is the model parameter of the encoder based on the modified model; Step 45: Identify whether the first loss value meets the preset first loss value range; if the first loss value meets the first loss value range, identify whether the current training record is the last question-and-answer text record of the first training set; if so, proceed to step 46; otherwise, take the next question-and-answer text record of the first training set as the new current training record and return to step 43; if the first loss value does not meet the first loss value range, based on the preset first model optimizer, move towards making the first model loss function L... M1 The model parameters of the basic modified model are modulated once in the direction that reaches the minimum value, and the process returns to step 43 when the current modulation ends. Here, the first loss value range in this embodiment of the invention is a pre-set numerical range; the first model optimizer includes the Adam optimizer and the SGD optimizer; Step 46, record the total number of records in the first evaluation set as The process iterates through all question-and-answer text records in the first evaluation set. During this iteration, the currently traversed question-and-answer text record is used as the current evaluation record. The question text of the current evaluation record is input as the corresponding query text X into the basic modified model for processing to obtain the corresponding generated text Y. The generated text Y and the answer text of the current evaluation record are then input into the basic modified model for processing, and the resulting two latent variables Z are... X The corresponding predicted text features and label text features are used as the basis for calculation. Then, using the cosine similarity algorithm, the similarity between the predicted and label text features corresponding to the current evaluation record is calculated to obtain the corresponding similarity d. j ; and at the end of this round of traversal, the obtained Similarity d j Substitute into the preset first model evaluation function L EV1The corresponding first evaluation value is obtained through calculation; Where 1 ≤ similarity index j ≤ ; Here, the first model evaluation function L in this embodiment of the invention EV1 Implemented based on the RMSE function, specifically as follows: ; Step 47: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 41; if it does, stop training and confirm that the training of the modified model of the basic modified model has ended.

[0043] Here, the first evaluation value range of this embodiment of the invention is a pre-set numerical range.

[0044] Step 5: After the modified model is trained, question-answer text pairs are assembled according to the preset target text library, and a second dataset is constructed based on the assembly results; then the first and second datasets are merged to obtain the corresponding third dataset; Specifically, this includes: Step 51, after the modified model is trained, question-answer text pairs are assembled according to the preset target text library and a second dataset is constructed based on the assembly results; Specifically, this includes: using each target text in the target text library as the current answer text; inputting the current answer text into a preset question conversion processing interface for processing, and using the output text of the question conversion processing interface as the corresponding current question text; combining the current question text and the current answer text to form a corresponding question-and-answer text record; and combining all the question-and-answer text records corresponding to the target text library to form the corresponding second dataset. Here, the target text library of this embodiment includes multiple target texts; the question conversion processing interface of this embodiment is a pre-set text task processing interface, which is used to provide a corresponding question for the current answer text input to the interface and output it; this interface can be another generative large language model, or an interface that converts the corresponding question text according to the input answer using a fixed template, or a manual processing interface; the second dataset of this embodiment also consists of multiple question and answer text records, which include question text and answer text; Step 52, and merge the first and second datasets to obtain the corresponding third dataset; Here, the third dataset in this embodiment of the invention also consists of multiple question-and-answer text records, which include question text and answer text.

[0045] Step 6: Based on the basic modified model, perform feature encoding on the target text library to obtain the corresponding feature vector library; introduce a retrieval unit and a feature aggregation module into the basic modified model to form a retrieval enhancement language model; and connect the retrieval enhancement language model with the feature vector library. Specifically, this includes: Step 61, performing feature encoding on the target text library based on the basic modified model to obtain the corresponding feature vector library; The feature vector library includes multiple text feature vectors, each corresponding one-to-one with the target text. Specifically, this includes: taking each target text in the target text library as the current text; inputting the current text as the corresponding query text X into the basic modified model for processing; and processing the latent variable Z obtained from this process. X These serve as the corresponding text feature vectors; and all the obtained text feature vectors form the corresponding feature vector library; Step 62, and introduce a retrieval tool and a feature aggregation module into the basic modified model to form a retrieval-enhanced language model; and connect the retrieval-enhanced language model with the feature vector library.

[0046] Here, the input and output of the retrieval enhancement language model in this embodiment of the invention are consistent with the basic model. For example... Figure 4 As shown in the schematic diagram of the retrieval enhancement language model provided in Embodiment 1 of the present invention, the retrieval enhancement language model of the present invention includes an encoder, a latent variable module, a retrieval unit, a feature aggregation module, a decoder, and a low-rank enhancer; wherein, the retrieval unit interfaces with the feature vector library and is used to perform similar feature retrieval on the feature vector library based on the output features of the latent variable module; the feature aggregation module is used to aggregate features from the output features of the latent variable module and the retrieval unit and input the aggregated features into the decoder for decoding.

[0047] In the retrieval-enhanced language model, the output of the encoder is connected to the input of the latent variable module; the output of the latent variable module is connected to the input of the retrieval module and the feature aggregation module, respectively; the output of the retrieval module is connected to the input of the feature aggregation module; the retrieval module is also interconnected with the feature vector library; the output of the feature aggregation module is connected to the input of the decoder and the low-rank enhancer, respectively; the decoder is also interconnected with the low-rank enhancer.

[0048] The encoder is used to perform Gaussian distribution feature encoding on the query text X to obtain the corresponding encoded feature vector E, which is then sent to the latent variable module; this is similar to the encoder in the basic modified model.

[0049] The latent variable module is used to construct the corresponding latent variable Z based on the encoded feature vector E. X Send to the retrieval and feature aggregation modules; this is similar to the latent variable module of the basic modified model.

[0050] The retrieval system of this invention is used to perform cosine similarity algorithm on the latent variable Z. X The vector similarity with each text feature vector in the feature vector library is calculated; and the K text feature vectors corresponding to the top K similarity scores, sorted from highest to lowest similarity, are denoted as the corresponding latent variables Z. k ; and each latent variable Z k The corresponding vector similarity is denoted as similarity d. k ; and by each latent variable Z k and its corresponding similarity d k Form a corresponding search record s k (Z k ,d k ); and from the obtained K retrieval records s k (Z k ,d k ) form the corresponding search record sequence {s k (Z k ,d k The result is sent to the feature aggregation module. The total number of searches, K, is a preset positive integer.

[0051] The feature aggregation module in this embodiment of the invention is used to retrieve record sequences {s} k (Z k ,d k )} and latent variable Z X Feature aggregation is performed to obtain the corresponding latent variable Z, which is then sent to the decoder and low-rank enhancer.

[0052] Here, the feature aggregation method of the latent variable Z in this embodiment of the invention is as follows: , , , ; Where τ is a preset temperature coefficient; w k For latent variable Z k The corresponding weighting coefficient; Z S The corresponding retrieval feature vector; concatenate is the vector concatenation function; Z con The corresponding feature concatenation vector is Z. con The vector length is the retrieval feature vector Z. S With latent variable Z X The sum of vector lengths; W Z B Z The weight matrix and bias vector corresponding to the feature aggregation module; the vector shape of the latent variable Z and the latent variable ZX Maintain consistency.

[0053] In the retrieval-enhanced language model, the decoder is used to decode the model in an autoregressive manner according to the Transformer framework. It uses a decoding network to decode the latent variable Z, and during the decoding process, it enhances the latent variable features of the decoded feature vector through a low-rank enhancer. Based on the feature-enhanced decoded feature vector, it performs text generation processing to obtain the corresponding generated text Y output; this is similar to the decoder of the basic modified model.

[0054] In the retrieval-enhanced language model, when the decoder uses the decoding network for decoding: the latent variable Z is used as the input feature vector H0 of the first-layer decoding module to decode and obtain the corresponding feature vector H1. The feature vector H1 is then input into the corresponding first-layer low-rank module in the low-rank enhancer for latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V1. The enhanced feature vector V1 is then used as the new feature vector H1 and input into the second-layer decoding module, and so on, until the feature vector H1 obtained from the Lth-layer decoding module is finally decoded. L The corresponding Lth-level low-rank module in the input low-rank enhancer is used to perform latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V. L And will strengthen the feature vector V L As the new feature vector H L and the new feature vector H L The decoded feature vector serves as a feature enhancement; this is similar to the decoder in the basic modified model.

[0055] In the retrieval-enhanced language model, each l-th layer low-rank module of the low-rank enhancer is used to decompose the latent variable Z into the feature vector H corresponding to the l-th layer decoding module according to the low-rank decomposition method. l The latent variable features are enhanced to obtain the corresponding enhanced feature vector V. l This is similar to the low-rank enhancer in the basic modified model.

[0056] Among them, the enhanced feature vector V l The calculation method is as follows: .

[0057] Step 7: Train the retrieval enhancement language model based on the third dataset; Specifically, it includes: Step 71, dividing the third dataset into two sub-datasets according to the preset second segmentation ratio, denoted as the corresponding second training set and second evaluation set; Here, the second segmentation ratio in this embodiment of the invention is a preset ratio parameter, such as 8:2; both the second training set and the second evaluation set consist of multiple question-and-answer text records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio; Step 72: Take the first question-and-answer text record of the second training set as the current training record; Step 73: Input the question text of the current training record as the corresponding query text X into the retrieval enhancement language model for processing; save the process data of this processing; and record the answer text of the current training record as the corresponding expected text Y. * ; and for the expected text Y * Perform word segmentation sequence conversion to obtain the corresponding word segmentation sequence { }; Here, 1 ≤ word segmentation index u ≤ N 2 N 2 For the current expected text Y * The corresponding total number of word segments; For the current expected text Y * The uth segmented text; Step 74: Input the process data of this model processing into the preset second model loss function L. M2 The corresponding second loss value is obtained through calculation; Here, the second model loss function L in this embodiment of the invention M2 Based on the second reconstruction loss function L RECON2 Second divergence loss function L KL2 Implementation, specifically: , , ; Where β2 is the balancing weight parameter; This is the preset start marker; Given the latent variable Z and the word segmentation sequence Under the given conditions, the u-th segmented text generated by the model is the expected text Y. * The uth segmented text The probability of θ3; θ3 is the set of model parameters for the decoder and low-rank enhancer of the retrieval-enhanced language model; D KL () represents the Kullback-Leibler divergence function; The prior distribution corresponding to the latent variable Z is a pre-defined standard Gaussian distribution; θ4 represents the posterior distribution corresponding to the latent variable Z; θ4 is the set of model parameters for the encoder and feature aggregation module of the retrieval-enhanced language model. Step 75: Identify whether the second loss value meets the preset second loss value range; if the second loss value meets the second loss value range, identify whether the current training record is the last question-and-answer text record of the second training set; if so, proceed to step 76; otherwise, take the next question-and-answer text record of the second training set as the new current training record and return to step 73; if the second loss value does not meet the second loss value range, based on the preset second model optimizer, move towards making the second model loss function L... M2 The direction that reaches the minimum value modulates the set of model parameters of the feature aggregation module, decoder and low-rank enhancer of the retrieval enhancement language model in one round, and returns to step 73 when the modulation of this round ends; Here, the second loss value range in this embodiment of the invention is a pre-set numerical range; the second model optimizer includes the Adam optimizer and the SGD optimizer; Step 76, record the total number of records in the second evaluation set as The second evaluation set is iterated through all question-and-answer text records. During this iteration, the currently iterated question-and-answer text record is used as the current evaluation record. The question text of the current evaluation record is input into the retrieval enhancement language model as the corresponding query text X to obtain the corresponding generated text Y. The generated text Y and the answer text of the current evaluation record are then input into the retrieval enhancement language model for processing, and the resulting two latent variables Z are... X The corresponding predicted text features and label text features are used as the basis for calculation. Then, using the cosine similarity algorithm, the similarity between the predicted and label text features corresponding to the current evaluation record is calculated to obtain the corresponding similarity d. v ; and at the end of this round of traversal, the obtained Similarity d v Substitute into the preset second model evaluation function L EV2 The corresponding second evaluation value is obtained through calculation; Where 1 ≤ similarity index v ≤ ; Here, the second model evaluation function L in this embodiment of the invention EV2 Implemented based on the RMSE function, specifically as follows: ; Step 77: Identify whether the second evaluation value meets the preset range of the second evaluation value; if not, return to step 71; if it does, stop training and confirm that the training of the enhanced language model has ended.

[0058] Here, the second evaluation value range in this embodiment of the invention is a pre-set numerical range.

[0059] Step 8: After the augmentation model training is completed, use the retrieval augmentation language model to process the natural language question answering task; Specifically, this includes: receiving natural language text input by the user as the current query text X, inputting it into the retrieval enhancement language model for processing to obtain the corresponding generated text Y, and then feeding it back to the current user.

[0060] Figure 5 This is a block diagram of a processing device for a retrieval enhancement language model based on a variational autoencoder architecture, provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 5 As shown, the device includes: a base model selection module 201, a modified model setting module 202, a first data acquisition module 203, a modified model training module 204, a second data acquisition module 205, a final model setting module 206, a final model training module 207, and a model application module 208.

[0061] The base model selection module 201 is used to select any generative large language model based on the variational autoencoder architecture and which has completed natural language pre-training and question answering task-oriented fine-tuning as the base model; the base model is used to encode the features of the query text X input to the model, generate the corresponding latent variables based on the encoded features, and decode the generated text according to the latent variables to obtain the corresponding generated text Y and output it; the base model includes an encoder, a latent variable module and a decoder.

[0062] The modified model setting module 202 is used to introduce the corresponding low-rank enhancer into the decoder of the basic model to obtain the corresponding basic modified model; the basic modified model includes an encoder, a latent variable module, a decoder and a low-rank enhancer; the low-rank enhancer is used to enhance the latent variable features of each layer output of the decoder through low-rank transformation.

[0063] The first data acquisition module 203 is used to collect big data from the public question and answer dataset to obtain the corresponding first dataset.

[0064] The modified model training module 204 trains the modified model based on the first dataset.

[0065] The second data acquisition module 205 is used to assemble question-and-answer text pairs according to a preset target text library after the modified model training is completed, and to construct a second dataset based on the assembly results; and to merge the first and second datasets to obtain the corresponding third dataset; the target text library includes multiple target texts.

[0066] The final model setup module 206 encodes the target text library based on the basic modified model to obtain the corresponding feature vector library; it introduces a retrieval unit and a feature aggregation module into the basic modified model to form a retrieval-enhanced language model; and it interfaces the retrieval-enhanced language model with the feature vector library. The feature vector library includes multiple text feature vectors, each corresponding one-to-one with the target text. The retrieval-enhanced language model includes an encoder, a latent variable module, a retrieval unit, a feature aggregation module, a decoder, and a low-rank enhancer. The retrieval unit interfaces with the feature vector library and is used to perform similar feature retrieval on the feature vector library based on the output features of the latent variable module. The feature aggregation module is used to aggregate the output features of the latent variable module and the retrieval unit and input the aggregated features into the decoder for decoding.

[0067] The final model training module 207 trains the retrieval enhancement language model based on the third dataset.

[0068] The model application module 208 is used to process natural language question answering tasks using the retrieval-enhanced language model after the augmented model has been trained.

[0069] The present invention provides a processing device for a retrieval enhancement language model based on a variational autoencoder architecture, which can execute the method steps in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0070] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the base model selection module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0071] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0072] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0073] Figure 6 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 6As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0074] exist Figure 6 The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include Non-Volatile Memory, such as at least one disk storage device.

[0075] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0076] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0077] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for processing a retrieval-enhanced language model based on a variational autoencoder architecture. As described above, this invention first uses any generative large language model implemented based on a VAE architecture, which has completed natural language pre-training and question-answering task-oriented fine-tuning, as a base model. This base model includes an encoder, a latent variable module, and a decoder. A low-rank enhancer is introduced into the base model to obtain a corresponding modified base model. Then, a first dataset is constructed by collecting publicly available question-answering datasets; and the modified model is trained based on the first dataset. After the modified model training is completed, question-answer text pairs are assembled from an external knowledge base (i.e., the target text library), and a second dataset is constructed based on the assembly results. The first and second datasets are merged to obtain a third dataset; and a feature vector library corresponding to the target text library is constructed using the modified base model. Then, a retrieval unit and a feature aggregation module are introduced into the modified model to form a retrieval-enhanced language model, which is then connected to the feature vector library; and the retrieval-enhanced language model is trained based on the third dataset. After the enhanced model training is completed, the final retrieval-enhanced language model is used to process natural language question-answering tasks. The retrieval enhancement language model implemented based on the VAE architecture of this invention improves the quality and timeliness of generated text through the retrieval engine, reduces feature noise, improves feature richness, and enhances generation quality and factual accuracy through the feature aggregation module, and prevents posterior collapse and improves generation quality through the low-rank enhancer.

[0078] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0079] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A processing method for a retrieval enhancement language model based on a variational autoencoder architecture, characterized in that, The method includes: Any generative large language model based on a variational autoencoder architecture, which has completed natural language pre-training and question-answering task-oriented fine-tuning, is used as the base model; the base model is used to encode the features of the query text X input to the model, generate corresponding latent variables based on the encoded features, and decode the generated text according to the latent variables to obtain the corresponding generated text Y and output it; the base model includes an encoder, a latent variable module, and a decoder. A corresponding low-rank enhancer is introduced into the decoder of the base model to obtain a corresponding base modified model; the base modified model includes the encoder, the latent variable module, the decoder and the low-rank enhancer; the low-rank enhancer is used to enhance the latent variable features of each layer output of the decoder through low-rank transformation; The first dataset was obtained by performing big data collection on the publicly available question-and-answer dataset. The modified model is trained based on the first dataset. After the modified model is trained, question-answer text pairs are assembled according to a preset target text library, and a second dataset is constructed based on the assembly results; then the first and second datasets are merged to obtain the corresponding third dataset; the target text library includes multiple target texts; Based on the aforementioned modified model, the target text library is feature-encoded to obtain a corresponding feature vector library. A retrieval unit and a feature aggregation module are introduced into the modified model to form a retrieval-enhanced language model. This retrieval-enhanced language model is then integrated with the feature vector library. The feature vector library includes multiple text feature vectors, each corresponding one-to-one with the target text. The retrieval-enhanced language model includes the encoder, the latent variable module, the retrieval unit, the feature aggregation module, the decoder, and the low-rank enhancer. The retrieval unit is integrated with the feature vector library and is used to perform similar feature retrieval on the feature vector library based on the output features of the latent variable module. The feature aggregation module is used to aggregate the output features of the latent variable module and the retrieval unit and input the aggregated features into the decoder for decoding. The retrieval enhancement language model is trained based on the third dataset. After the augmented model is trained, the retrieval augmented language model is used to process natural language question answering tasks.

2. The processing method for retrieval enhancement language models based on variational autoencoder architecture according to claim 1, characterized in that, The first, second, and third datasets each consist of multiple question-and-answer text records; these records include question text and answer text. The publicly available question-and-answer datasets include at least the WebQA dataset, the DuReader dataset, and the LCQMC dataset.

3. The processing method for the retrieval enhancement language model based on the variational autoencoder architecture according to claim 2, characterized in that, In the basic model, the input of the encoder is connected to the input of the latent variable module; the output of the latent variable module is connected to the input of the decoder. The encoder is implemented based on the encoding model of the Transformer framework; the encoder is used to perform Gaussian distribution feature encoding on the query text X to obtain the corresponding encoded feature vector E and send it to the latent variable module; Wherein, the encoded feature vector E is composed of the mean feature vector µ X Log-variance vector composition; σ is the variance feature vector of the query text X; X Let X be the standard deviation feature vector of the query text X; The latent variable module is used to construct the corresponding latent variable Z based on the encoded feature vector E. X Send to the decoder; Wherein, the latent variable Z X The construction method is as follows: ； ε is the Gaussian noise tensor; The decoder is implemented based on the Transformer framework decoding model; the decoding network of the decoder consists of L sequentially connected layer l decoding modules, where 1 ≤ layer index l ≤ L, and L is the total number of decoding modules in the decoder; the input and output features of the layer l decoding module are denoted as the corresponding feature vector H. l-1 H l ; In the base model, the decoder is used to decode the model in an autoregressive manner according to the Transformer framework, and the decoding network is used to decode the latent variable Z. X Perform feature decoding and text generation processing based on the decoded feature vectors to obtain the corresponding generated text Y output; In the basic model, when the decoder uses the decoding network for decoding: it converts the latent variable Z... X The input feature vector H0 of the first-layer decoding module is fed into the decoding network for layer-by-layer decoding, and the feature vector H output by the Lth-layer decoding module is... L This is the decoded feature vector output by the decoding network.

4. The processing method for retrieval enhancement language models based on variational autoencoder architecture according to claim 3, characterized in that, In the aforementioned basic modified model, the output of the encoder is connected to the input of the latent variable module; the output of the latent variable module is connected to the inputs of the decoder and the low-rank enhancer, respectively; the decoder is also interconnected with the low-rank enhancer. The encoder is used to perform Gaussian distribution feature encoding on the query text X to obtain the corresponding encoded feature vector E, which is then sent to the latent variable module. The latent variable module is used to construct the corresponding latent variable Z based on the encoded feature vector E. X Send to the decoder and the low-rank enhancer; In the aforementioned modified model, the decoder is used to decode the model's autoregressive decoding method according to the Transformer framework, and the decoding network is used to decode the latent variable Z. X Feature decoding is performed, and during the decoding process, the low-rank enhancer is used to enhance the latent variable features of the decoded feature vector. Based on the enhanced decoded feature vector, text generation processing is performed to obtain the corresponding generated text Y output. In the aforementioned basic modified model, when the decoder uses the decoding network for decoding: it converts the latent variable Z... X The input feature vector H0 of the first-layer decoding module is decoded to obtain the corresponding feature vector H1. This feature vector H1 is then input into the corresponding first-layer low-rank module in the low-rank enhancer for latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V1. This enhanced feature vector V1 is then used as the new feature vector H1 and input into the second-layer decoding module, and so on. Finally, the feature vector H0 obtained from the Lth-layer decoding module is... L The corresponding Lth-level low-rank module in the low-rank enhancer is input to perform latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V. L and the enhanced feature vector V L As the new feature vector H L and the new feature vector H L As the decoded feature vector for feature enhancement; The low-rank enhancer includes L low-rank modules of layer l; each of the low-rank modules of the low-rank enhancer is used to perform low-rank decomposition based on the latent variable Z. X The feature vector H corresponding to the l-th layer decoding module l The latent variable features are enhanced to obtain the corresponding enhanced feature vector V. l ; Wherein, the enhanced feature vector V l The calculation method is as follows: , R is the preset total number of ranks, where 1 ≤ rank index r ≤ R; The content path weight matrix of the l-th low-rank module is... Let r be the potential path weight matrix of the l-th low-rank module; , The r-th low-rank weight matrix component that makes up the l-th low-rank module; ⊙ is the Hadamard product.

5. The processing method for retrieval enhancement language models based on variational autoencoder architecture according to claim 4, characterized in that, In the retrieval-enhanced language model, the output of the encoder is connected to the input of the latent variable module; the output of the latent variable module is connected to the inputs of the retrieval unit and the feature aggregation module; the output of the retrieval unit is connected to the input of the feature aggregation module; the retrieval unit is also interconnected with the feature vector library; the output of the feature aggregation module is connected to the inputs of the decoder and the low-rank enhancer; the decoder is also interconnected with the low-rank enhancer. The encoder is used to perform Gaussian distribution feature encoding on the query text X to obtain the corresponding encoded feature vector E, which is then sent to the latent variable module. The latent variable module is used to construct the corresponding latent variable Z based on the encoded feature vector E. X Send to the retriever and the feature aggregation module; The retrieval unit is used to perform a cosine similarity algorithm on the latent variable Z. X The vector similarity with each of the text feature vectors in the feature vector library is calculated; and the K text feature vectors corresponding to the top K similarity scores in descending order of similarity are denoted as the corresponding latent variables Z. k ; and each of the aforementioned latent variables Z k The corresponding vector similarity is denoted as similarity d. k ; and by each of the aforementioned latent variables Z k and the corresponding similarity d k Form a corresponding search record s k (Z k ,d k ); and from the obtained K retrieved records s k (Z k ,d k ) form the corresponding search record sequence {s k (Z k ,d k Send the result to the feature aggregation module; the total number of searches, K, is a preset positive integer; The feature aggregation module is used to perform the retrieval record sequence {s} k (Z k ,d k )} and the latent variable Z X The corresponding latent variable Z is obtained by performing feature aggregation and sent to the decoder and the low-rank enhancer; The feature aggregation method for the latent variable Z is as follows: ， ， , ； τ is the preset temperature coefficient; w k For the latent variable Z k The corresponding weighting coefficient; Z S The corresponding retrieval feature vector; concatenate is the vector concatenation function; Z con The corresponding feature concatenation vector, Z con The vector length is the retrieved feature vector Z. S With the latent variable Z X The sum of vector lengths; W Z B Z The weight matrix and bias vector corresponding to the feature aggregation module; the vector shape of the latent variable Z and the latent variable Z X Maintain consistency; In the retrieval enhancement language model, the decoder is used to perform autoregressive decoding of the decoding model according to the Transformer framework. The decoding network is used to perform feature decoding on the latent variable Z. During the decoding process, the low-rank enhancer is used to enhance the latent variable features of the decoded feature vector. Based on the feature-enhanced decoded feature vector, text generation processing is performed to obtain the corresponding generated text Y output. In the retrieval enhancement language model, when the decoder uses the decoding network for decoding: the latent variable Z is used as the input feature vector H0 of the first-layer decoding module to decode and obtain the corresponding feature vector H1; the feature vector H1 is then input into the corresponding first-layer low-rank module in the low-rank enhancer for latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V1; the enhanced feature vector V1 is then used as the new feature vector H1 and input into the second-layer decoding module, and so on, until the feature vector H0 obtained by the Lth-layer decoding module is finally decoded. L The corresponding Lth-level low-rank module in the low-rank enhancer is input to perform latent variable feature enhancement processing to obtain the corresponding enhanced feature vector V. L and the enhanced feature vector V L As the new feature vector H L and the new feature vector H L As the decoded feature vector for feature enhancement; In the retrieval enhancement language model, each of the l-th layer low-rank modules of the low-rank enhancer is used to decompose the latent variable Z into the feature vector H corresponding to the l-th layer decoding module according to the low-rank decomposition method. l The latent variable features are enhanced to obtain the corresponding enhanced feature vector V. l ; Wherein, the enhanced feature vector V l The calculation method is as follows: 。 6. The processing method for retrieval enhancement language models based on variational autoencoder architecture according to claim 4, characterized in that, The step of training the modified model based on the first dataset specifically includes: Step 61: Divide the first dataset into two sub-datasets according to a preset first segmentation ratio, denoted as the first training set and the first evaluation set. Wherein, both the first training set and the first evaluation set consist of multiple question-and-answer text records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; Step 62: Take the first question-and-answer text record of the first training set as the current training record; Step 63: Input the question text from the current training record as the corresponding query text X into the base modified model for processing; save the process data of this processing; and record the answer text from the current training record as the corresponding expected text Y. * ; and for the desired text Y * Perform word segmentation sequence conversion to obtain the corresponding word segmentation sequence { }; Where 1 ≤ word segmentation index i ≤ N 1 N 1 The desired text Y is as described above. * The corresponding total number of word segments; The desired text Y is as described above. * The i-th segmented text; Step 64: Input the process data of this model processing into the preset first model loss function L. M1 The corresponding first loss value is obtained through calculation; Wherein, the first model loss function L M1 Based on the first reconstruction loss function L RECON1 and the first divergence loss function L KL1 Implementation, specifically: ， ， ; β1 is the balancing weight parameter; This is the preset start marker; Given the latent variable Z X and word segmentation sequence Under the given conditions, the i-th segmented text generated by the model is the expected text Y. * The i-th segmented text The probability; θ1 is the set of model parameters of the decoder and the low-rank enhancer of the basic modified model; D KL () represents the Kullback-Leibler divergence function; To be related to the latent variable Z X The corresponding prior distribution is a pre-defined standard Gaussian distribution; To be related to the latent variable Z X The corresponding posterior distribution; θ2 is the model parameter of the encoder in the basic modified model; Step 65: Identify whether the first loss value meets a preset first loss value range; if the first loss value meets the first loss value range, identify whether the current training record is the last question-and-answer text record in the first training set; if yes, proceed to step 66; otherwise, take the next question-and-answer text record in the first training set as the new current training record and return to step 63; if the first loss value does not meet the first loss value range, optimize the first model based on a preset first model optimizer to make the first model loss function L... M1 The model parameters of the basic modified model are modulated once in the direction that reaches the minimum value, and the process returns to step 63 when the modulation ends. The first model optimizer includes the Adam optimizer and the SGD optimizer. Step 66, record the total number of records in the first evaluation set as... The first evaluation set is iterated through all the question-and-answer text records in the first evaluation set. During this iteration, the currently iterated question-and-answer text record is used as the current evaluation record. The question text of the current evaluation record is used as the corresponding query text X and input into the basic modified model for processing to obtain the corresponding generated text Y. The generated text Y and the answer text of the current evaluation record are input into the basic modified model for processing, and the two latent variables Z obtained are... X The predicted text features and label text features are used as the corresponding text features; and the similarity d is calculated by using the cosine similarity algorithm to obtain the similarity between the predicted and label text features corresponding to the current evaluation record. j ; and at the end of this round of traversal, the obtained The similarity d j Substitute into the preset first model evaluation function L EV1 The corresponding first evaluation value is obtained through calculation; Where 1 ≤ similarity index j ≤ ; The first model evaluation function L EV1 Implemented based on the RMSE function, specifically as follows: ； Step 67: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 61; if it does, stop training and confirm that the training of the modified model of the basic modified model has ended.

7. The processing method for retrieval enhancement language models based on variational autoencoder architecture according to claim 2, characterized in that, The step of assembling question-and-answer text pairs based on a preset target text library and constructing a second dataset based on the assembly results specifically includes: Each target text in the target text library is used as the current answer text; the current answer text is input into a preset question conversion processing interface for processing, and the output text of the question conversion processing interface is used as the corresponding current question text; the current question text and the current answer text form a corresponding question and answer text record; and all the question and answer text records corresponding to the target text library form the corresponding second dataset; The question conversion processing interface is used to provide a corresponding question for the current answer text input to the interface and output it.

8. The processing method for retrieval enhancement language models based on variational autoencoder architecture according to claim 1, characterized in that, The step of performing feature encoding on the target text library based on the aforementioned modified model to obtain the corresponding feature vector library specifically includes: Each target text in the target text library is taken as the current text; and the current text is used as the corresponding query text X and input into the basic redesign model for processing, and the latent variable Z obtained from this processing is used as the current text. X As the corresponding text feature vector; and the corresponding feature vector library is composed of all the obtained text feature vectors.

9. The processing method for retrieval enhancement language models based on variational autoencoder architecture according to claim 5, characterized in that, The step of training the enhanced language model for retrieval based on the third dataset specifically includes: Step 91: Divide the third dataset into two sub-datasets according to the preset second segmentation ratio, and denoted as the corresponding second training set and second evaluation set; The second training set and the second evaluation set each consist of multiple question-and-answer text records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio. Step 92: Take the first question-and-answer text record of the second training set as the current training record; Step 93: Input the question text from the current training record as the corresponding query text X into the retrieval enhancement language model for processing; save the process data of this processing; and record the answer text from the current training record as the corresponding expected text Y. * ; and for the desired text Y * Perform word segmentation sequence conversion to obtain the corresponding word segmentation sequence { }; Where 1 ≤ word segmentation index u ≤ N 2 N 2 The desired text Y is as described above. * The corresponding total number of word segments; The desired text Y is as described above. * The uth segmented text; Step 94: Input the process data of this model processing into the preset second model loss function L. M2 The corresponding second loss value is obtained through calculation; Wherein, the second model loss function L M2 Based on the second reconstruction loss function L RECON2 Second divergence loss function L KL2 Implementation, specifically: ， ， ; β2 is the balancing weight parameter; This is the preset start marker; Given the latent variable Z and the word segmentation sequence Under the given conditions, the u-th segmented text generated by the model is the expected text Y. * The uth segmented text The probability of θ3; θ3 is the set of model parameters of the decoder and the low-rank enhancer of the retrieval enhancement language model; D KL () represents the Kullback-Leibler divergence function; The prior distribution corresponding to the latent variable Z is specifically a pre-defined standard Gaussian distribution; θ4 represents the posterior distribution corresponding to the latent variable Z; θ4 is the set of model parameters for the encoder and the feature aggregation module of the retrieval enhancement language model. Step 95: Identify whether the second loss value meets the preset second loss value range; if the second loss value meets the second loss value range, identify whether the current training record is the last question-and-answer text record of the second training set; if yes, proceed to step 96; otherwise, take the next question-and-answer text record of the second training set as the new current training record and return to step 93; if the second loss value does not meet the second loss value range, based on the preset second model optimizer, move towards making the second model loss function L... M2 The direction that reaches the minimum value is used to modulate the model parameter set of the feature aggregation module, the decoder and the low-rank enhancer of the retrieval enhancement language model in one round, and the process returns to step 93 when the modulation in this round ends. The second model optimizer includes the Adam optimizer and the SGD optimizer; Step 96, record the total number of records in the second evaluation set as... The second evaluation set is iterated through all the question-and-answer text records in the second evaluation set. During this iteration, the currently iterated question-and-answer text record is used as the current evaluation record. The question text of the current evaluation record is used as the corresponding query text X and input into the retrieval enhancement language model for processing to obtain the corresponding generated text Y. The generated text Y and the answer text of the current evaluation record are input into the retrieval enhancement language model for processing, and the two latent variables Z obtained are processed. X The predicted text features and label text features are used as the corresponding text features; and the similarity d is calculated by using the cosine similarity algorithm to obtain the similarity between the predicted and label text features corresponding to the current evaluation record. v ; and at the end of this round of traversal, the obtained The similarity d v Substitute into the preset second model evaluation function L EV2 The corresponding second evaluation value is obtained through calculation; Where 1 ≤ similarity index v ≤ ; The second model evaluation function L EV2 Implemented based on the RMSE function, specifically as follows: ； Step 97: Identify whether the second evaluation value meets the preset second evaluation value range; if not, return to step 91; if yes, stop training and confirm that the training of the enhanced language model for retrieval has ended.

10. An apparatus for performing the processing method of the retrieval enhancement language model based on the variational autoencoder architecture as described in any one of claims 1-9, characterized in that, The device includes: a base model selection module, a modified model setting module, a first data acquisition module, a modified model training module, a second data acquisition module, a final model setting module, a final model training module, and a model application module; The base model selection module is used to select any generative large language model based on a variational autoencoder architecture that has completed natural language pre-training and question-answering task-oriented fine-tuning as the base model; the base model is used to encode the features of the query text X input to the model, generate corresponding latent variables based on the encoded features, and decode the generated text according to the latent variables to obtain the corresponding generated text Y and output it; the base model includes an encoder, a latent variable module and a decoder. The modified model setting module is used to introduce a corresponding low-rank enhancer to the decoder of the base model to obtain a corresponding base modified model; the base modified model includes the encoder, the latent variable module, the decoder and the low-rank enhancer; the low-rank enhancer is used to enhance the latent variable features of each layer output of the decoder through low-rank transformation; The first data acquisition module is used to perform big data collection on the public question and answer dataset to obtain the corresponding first dataset; The modified model training module trains the modified model based on the first dataset on the basic modified model; The second data acquisition module is used to assemble question-and-answer text pairs according to a preset target text library after the modified model training is completed, and to construct a second dataset based on the assembly results; and to merge the first and second datasets to obtain the corresponding third dataset; the target text library includes multiple target texts; The final model setting module performs feature encoding on the target text library based on the basic modified model to obtain a corresponding feature vector library; and introduces a retrieval unit and a feature aggregation module into the basic modified model to form a retrieval-enhanced language model; and interfaces the retrieval-enhanced language model with the feature vector library; the feature vector library includes multiple text feature vectors, each corresponding one-to-one with the target text; the retrieval-enhanced language model includes the encoder, the latent variable module, the retrieval unit, the feature aggregation module, the decoder, and the low-rank enhancer; the retrieval unit interfaces with the feature vector library and is used to perform similar feature retrieval on the feature vector library based on the output features of the latent variable module; the feature aggregation module is used to aggregate features from the output features of the latent variable module and the retrieval unit and input the aggregated features into the decoder for decoding; The final model training module trains the retrieval enhancement language model based on the third dataset. The model application module is used to process natural language question answering tasks using the retrieval-enhanced language model after the enhanced model training is completed.

11. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-9; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-9.

Images