A processing method and device of a dense encoder based on a large language model

Abstract

This invention relates to a processing method and apparatus for a dense encoder based on a large language model. The method includes: selecting a large language model that has been pre-trained and fine-tuned for NLP tasks and implemented based on a pure decoder architecture as the target model; obtaining a bidirectional encoder by solidifying the causal mask matrix used by the target model's decoder during inference into an all-one matrix; and forming a dense encoder by sequentially connecting the target model's embedding encoding module and multiple bidirectional encoders; performing a first-stage fine-tuning of the dense encoder using a masked word prediction task; performing a second-stage fine-tuning of the dense encoder using an unsupervised contrastive learning mechanism; after fine-tuning, constructing a document vector library for a user-specified target document library using the dense encoder; and providing retrieval services for the target document library based on the document vector library and the dense encoder. Using the dense encoder of this invention to process text retrieval tasks can improve retrieval accuracy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a processing method and apparatus for a dense encoder based on a large language model. Background Technology

[0002] Large Language Models (LLMs) implemented based on the decoder (Decoder-A) framework of the Transformer model are also known as Decoder-only LLMs. These large language models are the current mainstream autoregressive generative models, such as the GPT series models, LLaMA series models, OPT series models, GLM series models, DeepSeek series models, etc.

[0003] The decoder module of Decoder-only LLMs is denoted as Decoder-B. The difference between Decoder-A and Decoder-B is that Decoder-B does not contain the multi-head attention layer (MHA) that performs cross-attention operations with the encoder output in Decoder-A, nor its corresponding residual connections and layer normalization units (Add&Norm). In other words, Decoder-A is composed of a masked multi-head attention layer (MMHA) + Add&Norm, MHA + Add&Norm, and a feedforward neural network (FNN) + Add&Norm connected sequentially, while Decoder-B is simply composed of MMHA + Add&Norm and FNN + Add&Norm connected sequentially. Otherwise, the decoding logic of the decoder network (composed of multiple Decoder-B connections) in Decoder-only LLMs is consistent with the decoding logic of the Transformer model.

[0004] The decoder vectors output by the decoder network of Decoder-only LLMs, after pre-training and fine-tuning for Natural Language Processing (NLP) tasks, are well-suited for a range of autoregressive prediction tasks, such as next-word prediction and sequence-to-sequence prediction. However, these decoder vectors are insufficient as high-dimensional semantic representation vectors (dense vectors) of the input text. This is because, according to the decoding logic of the Transformer model, the masked multi-head attention layer in Decoder-B uses a set mask vector (also known as a causal mask vector) to ensure that the model only performs calculations in a single-line direction (from left to right or from right to left) when performing Scaled Dot-Product Attention calculations. This results in the final output decoder vector having sub-vectors corresponding to each word that are only related to the preceding text (also known as the context) of the current word segmentation position and not related to the following text (also known as the context). In other words, although the decoder vectors output by the decoder network of Decoder-only LLMs after pre-training and fine-tuning for NLP tasks can be well used for a series of autoregressive prediction tasks, they lack bidirectional encoding capabilities when used as dense vectors of input text.

[0005] The technical problem this invention aims to solve is how to modify and fine-tune a decoder of a pre-trained and NLP task-fine-tuned Decoder-only LLM to make it a dense vector encoder with bidirectional encoding capabilities (denoted as Encoder-B). Because Encoder-B originates from a trained autoregressive model, it can not only perform bidirectional encoding during the encoding process but also inherit the long-range dependencies learned by the original model during training—a capability not possessed by traditional encoders based on the Transformer model encoder structure (denoted as Encoder-A). In principle, compared to Encoder-A, in retrieval scenarios, Encoder-B's long-range dependency capture and bidirectional encoding capabilities should better understand the overall content and structure of a document and generate more expressive dense text vectors. Retrieval based on the dense text vectors output by Encoder-B can further improve retrieval accuracy and recall. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a processing method, apparatus, electronic device, and computer-readable storage medium for a dense encoder based on a large language model. This invention first selects a large language model that has completed pre-training and NLP task fine-tuning and is implemented based on a pure decoder architecture as the corresponding target model. The decoder of the target model is then used as the corresponding target decoder. A bidirectional encoder is obtained by modifying the causal mask matrix used by the target decoder during inference by fixing it into an all-one matrix. Multiple bidirectional encoders are sequentially connected to form a corresponding bidirectional encoding network. The embedding encoding module of the target model and the bidirectional encoding network are then connected to form a dense encoder. Next, a corresponding model training framework is constructed for the dense encoder, and a first-stage fine-tuning is performed using the masked word prediction task within this model training framework. After the first-stage fine-tuning, a second-stage fine-tuning is performed using an unsupervised contrastive learning mechanism. After the second-stage fine-tuning, a user-specified document library is used as the target document library. A document vector library is constructed for the target document library using the dense encoder, and a retrieval service is provided for the target document library based on the document vector library and the dense encoder. This invention modifies the decoder of the target model to obtain a dense encoder with bidirectional encoding capabilities. A first-stage fine-tuning ensures that the dense encoder can successfully inherit the ability to capture long-range dependencies, and a second-stage fine-tuning further enhances the semantic representation capabilities of the dense encoder. Using the dense encoder provided by this invention to process text retrieval tasks can effectively improve retrieval accuracy and recall.

[0007] To achieve the above objectives, a first aspect of the present invention provides a processing method for a dense encoder based on a large language model, the method comprising:

[0008] A large language model that has been pre-trained and fine-tuned for NLP tasks and implemented based on a pure decoder architecture is selected as the corresponding target model; the decoder of the target model is selected as the corresponding target decoder; a bidirectional encoder is obtained by modifying the causal mask matrix used by the target decoder during inference by fixing it into an all-1 matrix; multiple bidirectional encoders are sequentially connected to form a corresponding bidirectional encoding network, and the embedding encoding module of the target model and the bidirectional encoding network are connected to form a dense encoder; the NLP task includes at least text generation, information extraction, and question answering tasks;

[0009] A corresponding model training framework is constructed for the dense encoder; and the dense encoder is fine-tuned in one stage through the masked word prediction task of the model training framework; after the first stage of fine-tuning is completed, the dense encoder is fine-tuned in two stages through an unsupervised contrastive learning mechanism.

[0010] After the second-stage fine-tuning is completed, the user-specified document library is used as the target document library; and a document vector library is constructed for the target document library using the dense encoder; and a retrieval service is provided for the target document library based on the document vector library and the dense encoder; the target document library includes multiple first documents; the document vector library includes multiple first dense vectors, and each first dense vector corresponds one-to-one with a first document.

[0011] Preferably, the dense encoder is used to perform dense vector encoding processing on the input text of the encoder and output the corresponding text dense vector Y;

[0012] The dense encoder is composed of the embedded coding module and the bidirectional coding network connected together;

[0013] The embedding encoding module is used to identify whether the input text contains the preset masking word "[MASK]". If it does not contain the preset masking word, the input text is segmented according to the word segmentation rules of the target model and the preset vocabulary of the target model to obtain the corresponding current word segmentation sequence. If it contains the preset masking word, the input text is segmented into multiple text segments without the preset masking word, using each preset masking word as a cutting position. Each text segment is then segmented according to the word segmentation rules of the target model and the preset vocabulary to obtain the corresponding segment word segmentation sequence. The embedded encoding module is used to identify whether the input text contains the preset masking word "[MASK]". Let the masking word be an independent masking word, and sort the segmented words of all the segmented word sequences and all the masking words according to the order in the input text to obtain the corresponding current segmented word sequence; add a preset starting word "[SOS]" at the beginning position of the current segmented word sequence, and take the current segmented word sequence with the added starting word as the corresponding text segmented word sequence W; and perform embedding encoding processing on the text segmented word sequence W according to the embedding encoding rules of the target model to obtain the corresponding text embedding vector X, which is sent to the bidirectional coding network; the text segmented word sequence W consists of multiple words w i Sort by, 1 ≤ index i ≤ N W N W The total number of words in the text segmentation sequence; the text embedding vector X is composed of N W Each word segmentation embedding vector x i Composed of the word segmentation embedding vector x i With the word segmentation w i One-to-one correspondence; when the input text contains the preset blocked words, the total number of the preset blocked words is denoted as N. M The word segmentation embedding vector x corresponding to each of the preset masking words i Let be the corresponding masking word embedding vector. 1≤indexj≤NM ;

[0014] The bidirectional coding network consists of N X The bidirectional encoders are connected sequentially, and the total number of encoders is N. X It is a preset positive integer greater than 1; the bidirectional encoding network is used to pass through the internal N X The bidirectional encoder performs continuous bidirectional encoding on the text embedding vector X to obtain the corresponding dense text vector Y and outputs it; the dense text vector Y is composed of N... W Dense vector y for word segmentation i Composed of the word segmentation dense vector y i With the word segmentation embedding vector x i One-to-one correspondence; total number N M When it is not zero, it is related to the embedding vectors of each of the aforementioned masking words. The corresponding word segmentation dense vector y i Let be the dense vector of the corresponding masking word.

[0015] The bidirectional encoder and the target decoder have the same model structure, both consisting of a masked multi-head attention layer, a residual connection and layer normalization unit, a feedforward neural network, and another residual connection and layer normalization unit connected sequentially.

[0016] The inference process of the masked multi-head attention layer of each of the bidirectional encoders is as follows:

[0017]

[0018] 1 ≤ index n ≤ N X ;X n-1 X n These are the input and output vectors of the nth bidirectional encoder, respectively; d n-1 For the input vector X n-1 Feature dimensions; Q n K n V n These are the query vector, key vector, and value vector corresponding to the nth bidirectional encoder; The query vector Q is respectively n The key vector K n and the value vector V n The corresponding weights; Let K be the key vector. n The inverted vector of M; all-ones It is an all-one matrix; the parameter set G of each of the bidirectional encoders n By corresponding weights composition;

[0019] The only difference between the inference process of the bidirectional encoder and the target decoder is that the masked multi-head attention layer of the bidirectional encoder uses an all-one matrix M during inference. all-ones The target decoder's masked multi-head attention layer uses a causal mask matrix M during inference. Causal The decoding order of the target model is the causal mask matrix M corresponding to word-by-word decoding from left to right. Causal The mask matrix is ​​a matrix with all 1s in the lower left triangle and all 0s in the upper right triangle. The decoding order is the same as the causal mask matrix M corresponding to word-by-word decoding from right to left. Causal It is a mask matrix with all 1s in the lower right triangle and all 0s in the upper left triangle.

[0020] Preferably, the model training framework is used to predict the original word text that is masked by the preset masking word in the masked text input to the framework and output the corresponding original word prediction sequence;

[0021] The blocked text is a string without punctuation marks and special characters, which contains at least one of the preset blocked words but is not entirely composed of the preset blocked words;

[0022] The original word prediction sequence consists of one or more predicted original word texts; the predicted original word texts in the original word prediction sequence correspond one-to-one with the preset blocked words in the blocked text; the order of all the predicted original word texts in the original word prediction sequence matches the order of all the preset blocked words in the blocked text;

[0023] The model training framework is composed of a dense encoder, a linear layer, a softmax function layer, and an output module connected sequentially.

[0024] The dense encoder is used to take the masked text as the corresponding input text, perform dense vector encoding on the input text, and output the corresponding dense vector Y of the text to be sent to the linear layer;

[0025] The linear layer is used to process each of the word segmentation dense vectors y in the text dense vector Y. i The word segmentation feature vector h is obtained by performing word list feature vector transformation. i ; and from all the obtained word segmentation feature vectors h i The ordered sorted vector sequence H is sent to the Softmax function layer; the word segmentation feature vector h i With the word segmentation dense vector y i One-to-one correspondence; and dense vectors of each of the aforementioned masked words. The corresponding word segmentation feature vector h i The previous word segmentation feature vector h i-1Let be the corresponding preceding word feature vector.

[0026] The Softmax function layer is used to determine the feature vectors of each preceding word. Perform vocabulary probability distribution calculations and use the results as the corresponding masked word probability vectors. And from the obtained probability vectors of all the masked words The sequentially ordered vector sequence P is sent to the output module; each of the masked word probability vectors Each is composed of multiple word segmentation probabilities, and the word segmentation probabilities correspond one-to-one with the word segmentation in the preset word list;

[0027] The output module is used to process the probability vectors of each of the masked words. The word segmentation within the table corresponding to the maximum word segmentation probability is used as the corresponding predicted original word text; and the original word prediction sequence is formed by sequentially sorting all the obtained predicted original word texts and outputting it.

[0028] Preferably, the step of fine-tuning the dense encoder through the masked word prediction task of the model training framework specifically includes:

[0029] Step 41: Randomly extract multiple sentences from the historical text processed by the target model, and take each extracted sentence as a corresponding first extracted text; and delete punctuation marks and special characters in each first extracted text;

[0030] Step 42, feed the parameter set G of each of the bidirectional encoders of the model training framework. n The corresponding parameter set is obtained by implanting a low-rank matrix.

[0031] Wherein, the parameter set By weight composition;

[0032] The masked multi-head attention layer of the bidirectional encoder is based on the parameter set. The reasoning process is as follows:

[0033]

[0034] For the weight The corresponding pair of low-rank matrices, For the weight The corresponding pair of low-rank matrices, For the weight The corresponding pair of low-rank matrices; the parameter set The corresponding low-rank matrix Form a corresponding low-rank matrix parameter set

[0035] Step 43: Initialize the current occlusion rate to the preset first occlusion rate;

[0036] Where 0 < first occlusion rate < 1;

[0037] Step 44: Based on the word segmentation rules of the target model and according to the preset vocabulary, pre-segment each of the first extracted texts to obtain the corresponding pre-segmented word sequences; and use the preset masking words to randomly replace the word segments of each of the pre-segmented word sequences according to the current masking rate to obtain the corresponding replaced word segments; and concatenate all the word segments of each of the replaced word segments to obtain the corresponding first training text; and count the total number of the preset masking words in all the first training texts to obtain the corresponding total number N. K ;

[0038] Wherein, the sequence segmentation of the replaced word segmentation sequence contains the preset masked words, and the total number of the preset masked words is the integer value of the product of the total number of sequence segmentation words and the current masking rate.

[0039] Step 45: Input each of the first training texts as the corresponding masked texts into the model training framework for prediction processing, and use the probability vectors of each masked word calculated by the Softmax function layer in this processing to perform prediction processing. Let p be a corresponding prediction vector. k 1 ≤ index k ≤ N K ; and based on each of the prediction vectors p k Set a corresponding label vector for the corresponding original word text. And by each of the predicted vectors p k and the corresponding label vector Form a corresponding first prediction-label pair

[0040] Wherein, the label vector The tag vector consists of multiple tag segmentation probabilities, each corresponding one-to-one with the segmentation probability within the preset vocabulary; Only one of the tags has a segmentation probability of 1, while the segmentation probabilities of the rest are all 0; the tag vector The word segmentation probability of the tag with a value of 1 corresponds to the word segmentation in the table and is matched with the corresponding original word text.

[0041] Step 46, obtain N K The first prediction-label pair Substitute the preset first model loss function L1 into the calculation to obtain the corresponding first loss value;

[0042] Wherein, the loss function L1 of the first model is:

[0043]

[0044] Step 47: Identify whether the first loss value meets the preset first loss value range; if it does, proceed to step 48; if not, identify whether the current occlusion rate is the first occlusion rate; if so, optimize all the low-rank matrix parameter sets of the bidirectional coding network in the direction that minimizes the first model loss function L1, based on the preset first model optimizer. And the linear layer parameters of the linear layer are modulated once; otherwise, based on a preset second model optimizer, the parameters of all the low-rank matrix are adjusted in the direction that minimizes the first model loss function L1. Perform one round of modulation, and after this round of modulation is completed, return to step 45 to continue training;

[0045] The first and second model optimizers each include at least the Adam optimizer and the SGD optimizer;

[0046] Step 48: Identify whether the current occlusion rate is the first occlusion rate; if yes, reset the current occlusion rate to the preset second occlusion rate and return to step 44 to continue training after resetting; if no, stop training and confirm that the first stage of fine-tuning is over.

[0047] Where 0 < first occlusion rate < second occlusion rate < 1.

[0048] Preferably, the two-stage fine-tuning of the dense encoder through an unsupervised contrastive learning mechanism specifically includes:

[0049] Step 51: Randomly extract multiple sentences from the historical text processed by the target model, and treat each extracted sentence as a corresponding second extracted text; delete punctuation marks and special characters in each second extracted text; and count the total number of second extracted texts to obtain the corresponding total number N. U ;

[0050] Step 52: Input each of the second extracted texts as the corresponding input texts into the dense encoder for dense vector encoding processing, and record the text dense vector Y output by this processing as the corresponding first sample;

[0051] Step 53: Each of the second extracted texts is used as the corresponding input text and input again into the dense encoder for dense vector encoding processing. During this processing, multiple residual connection and layer normalization units are randomly selected from the bidirectional encoding network as corresponding random scrambling units. At the end of the inference process of each random scrambling unit, a portion of the current unit's output vector is randomly scrambled, and the scrambled unit's output vector is passed downstream. The text dense vector Y finally output by this dense vector encoding processing is recorded as the corresponding second sample.

[0052] The random scrambling method is as follows: setting the selected sub-vector to an all-zero vector or adding Gaussian noise to the selected sub-vector;

[0053] Step 54, denote any one of the first samples as the corresponding sample s. u 1 ≤ index u ≤ N U ; and will be combined with each of the aforementioned samples s u The other different first samples are denoted as the current sample s. u negative samples 1≤index v≤N U -1; and by each of the aforementioned samples s u The corresponding N U -1 of the negative samples Form a corresponding negative sample set And with each of the aforementioned samples s u The corresponding second sample is denoted as the current sample s. u Positive samples And by each of the aforementioned samples s u and the corresponding positive samples With negative sample set Form a corresponding sample data group

[0054] Step 55, obtain N U The sample data groups The corresponding second loss value is obtained by substituting the preset second model loss function L2 into the calculation.

[0055] Wherein, the loss function L2 of the second model is:

[0056]

[0057] λ is a preset adjustment parameter; sim() is the cosine similarity function of the vectors. For the sample s u With the corresponding positive sample cosine similarity, For the sample s u With the corresponding negative sample Cosine similarity;

[0058] Step 56: Identify whether the second loss value meets the preset second loss value range; if not, then based on the preset third model optimizer, optimize all low-rank matrix parameter sets of the dense encoder in the direction that minimizes the second model loss function L2. Perform one round of modulation, and return to step 52 to continue training after this round of modulation is completed; if the conditions are met, stop training and confirm that the second-stage fine-tuning is complete;

[0059] The third model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0060] Preferably, the step of constructing a document vector library for the target document library using the dense encoder specifically includes:

[0061] Each of the first documents in the target document library is used as the corresponding input text and input into the dense encoder for dense vector encoding processing. The dense vector of the text output in this processing is used as a corresponding first dense vector. All the obtained first dense vectors are used to form the corresponding document vector library.

[0062] Preferably, providing retrieval services for the target document library based on the document vector library and the dense encoder specifically includes:

[0063] Each user-inputted query text is used as the current query text; the current query text is used as the input text and input into the dense encoder for dense vector encoding processing, and the text dense vector output by the current processing is used as the current dense vector; the cosine similarity between the current dense vector and each of the first dense vectors in the document vector library is calculated to obtain the corresponding first similarity; the first dense vectors whose first similarity exceeds a preset similarity threshold are recorded as the corresponding retrieval vectors; when the total number of retrieval vectors is not zero, the first documents in the target document library corresponding to each of the retrieval vectors are recorded as the corresponding retrieval documents; and all the obtained retrieval documents are sorted in descending order of the first similarity to form a corresponding retrieval document sequence and fed back to the current user.

[0064] A second aspect of the present invention provides an apparatus for implementing the processing method of a dense encoder based on a large language model as described in the first aspect above. The apparatus includes: a model modification module, a model fine-tuning module, and a retrieval application module.

[0065] The model modification module is used to select a large language model that has been pre-trained and fine-tuned for NLP tasks and implemented based on a pure decoder architecture as the corresponding target model; and to use the decoder of the target model as the corresponding target decoder; and to obtain a bidirectional encoder by modifying the causal mask matrix used by the target decoder during inference to a matrix of all 1s; and to form a corresponding bidirectional encoding network by sequentially connecting multiple bidirectional encoders, and to form a dense encoder by connecting the embedding encoding module of the target model and the bidirectional encoding network; the NLP task includes at least a text generation task, an information extraction task, and a question answering task;

[0066] The model fine-tuning module is used to construct a corresponding model training framework for the dense encoder; and to perform a first-stage fine-tuning of the dense encoder through the masked word prediction task of the model training framework; after the first-stage fine-tuning is completed, a second-stage fine-tuning of the dense encoder is performed through an unsupervised contrastive learning mechanism.

[0067] The retrieval application module is used to, after the two-stage fine-tuning is completed, take the user-specified document library as the target document library; and use the dense encoder to construct a document vector library for the target document library; and provide retrieval services for the target document library based on the document vector library and the dense encoder; the target document library includes multiple first documents; the document vector library includes multiple first dense vectors, and the first dense vectors correspond one-to-one with the first documents.

[0068] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0069] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0070] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0071] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0072] This invention provides a processing method, apparatus, electronic device, and computer-readable storage medium for a dense encoder based on a large language model. As described above, this invention first selects a large language model that has completed pre-training and NLP task fine-tuning and is implemented based on a pure decoder architecture as the corresponding target model. The decoder of the target model is then used as the corresponding target decoder. A bidirectional encoder is obtained by solidifying the causal mask matrix used by the target decoder during inference into an all-one matrix. Multiple bidirectional encoders are sequentially connected to form a corresponding bidirectional encoding network. The embedding encoding module of the target model and the bidirectional encoding network are then connected to form a dense encoder. Next, a corresponding model training framework is constructed for the dense encoder, and a first-stage fine-tuning is performed using the masked word prediction task of this model training framework. After the first-stage fine-tuning, a second-stage fine-tuning is performed using an unsupervised contrastive learning mechanism. After the second-stage fine-tuning, a user-specified document library is used as the target document library. A document vector library is constructed for the target document library using the dense encoder, and a retrieval service is provided for the target document library based on the document vector library and the dense encoder. This invention modifies the decoder of the target model to obtain a dense encoder with bidirectional encoding capabilities. A first-stage fine-tuning ensures that the dense encoder can successfully inherit the ability to capture long-range dependencies, and a second-stage fine-tuning further enhances the semantic representation capabilities of the dense encoder. The dense encoder of this invention effectively improves the retrieval accuracy and recall rate for document retrieval tasks. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of a dense encoder based on a large language model provided in Embodiment 1 of the present invention;

[0074] Figure 2 This is a module structure diagram of Decoder-A / B provided in Embodiment 1 of the present invention;

[0075] Figure 3 This is a block diagram of the model training framework provided in Embodiment 1 of the present invention;

[0076] Figure 4 This is a module structure diagram of a processing device based on a large language model for a dense encoder provided in Embodiment 2 of the present invention;

[0077] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0079] Embodiment 1 of the present invention provides a processing method for a dense encoder based on a large language model, such as... Figure 1 The schematic diagram shows a dense encoder based on a large language model provided in Embodiment 1 of the present invention. The method mainly includes the following steps:

[0080] Step 1: Select a large language model that has been pre-trained and fine-tuned for NLP tasks and implemented based on a pure decoder architecture as the corresponding target model; use the decoder of the target model as the corresponding target decoder; obtain a bidirectional encoder by solidifying the causal mask matrix used by the target decoder in the inference process into an all-1 matrix; connect multiple bidirectional encoders in sequence to form a corresponding bidirectional encoding network; and connect the embedding encoding module of the target model and the bidirectional encoding network to form a dense encoder.

[0081] Here, the target model in this embodiment of the invention is any large language model that has been pre-trained and fine-tuned for NLP tasks and implemented based on a pure decoder architecture, such as the GPT series models, LLaMA series models, OPT series models, GLM series models, DeepSeek series models, etc. The NLP tasks of this target model include at least text generation tasks, information extraction tasks, question answering tasks, text classification tasks, text translation tasks, etc.

[0082] As explained in the background description above, the target decoder in this embodiment of the invention is Decoder-B, mentioned in the background description above. For example... Figure 2 As shown in the module structure diagram of Decoder-A / B provided in Embodiment 1 of the present invention, the structural difference between Decoder-B and the decoder (Decoder-A) of the Transformer model is that Decoder-B does not contain a multi-head attention layer that performs cross-attention operations with the output of the Transformer encoder, as well as the residual connection and layer normalization unit corresponding to the attention layer.

[0083] The dense encoder of this invention is used to perform dense vector encoding processing on the input text of the encoder and output the corresponding dense text vector Y. This dense encoder consists of an embedded encoding module and a bidirectional encoding network connected together, as shown below. Figure 3The module structure diagram of the model training framework provided in Embodiment 1 of the present invention is shown.

[0084] The embedding encoding module of the dense encoder in this embodiment of the invention is used to identify whether the input text contains the preset masking word "[MASK]". If it does not contain the preset masking word, the input text is segmented according to the word segmentation rules of the target model and the preset vocabulary of the target model to obtain the corresponding current word segmentation sequence. If it contains the preset masking word, the input text is segmented into multiple text segments without the preset masking word at each preset masking word as the cutting position. Each text segment is segmented according to the word segmentation rules of the target model and the preset vocabulary to obtain the corresponding segment word segmentation sequence. Each preset masking word is treated as an independent masking word. The segment word segments of all segment word segments and all masking words are sorted according to the order of arrangement in the input text to obtain the corresponding current word segmentation sequence. A preset starting word "[SOS]" is added to the beginning position of the current word segmentation sequence. The current word segmentation sequence with the added starting word is taken as the corresponding text word segmentation sequence W. The text word segmentation sequence W is embedded according to the embedding encoding rules of the target model to obtain the corresponding text embedding vector X, which is sent to the bidirectional coding network.

[0085] Here, the preset vocabulary in this embodiment of the invention is the large language model vocabulary accumulated by the target model through pre-training and fine-tuning via NLP tasks; the text segmentation sequence W in this embodiment of the invention consists of multiple segmented words w i Sort by, 1 ≤ index i ≤ N W N W The total number of words in the text segmentation sequence; the text embedding vector X is composed of N. W Each word segmentation embedding vector x i Composition, word segmentation embedding vector x i With the word segment w i One-to-one correspondence; when the input text contains preset blocked words, the total number of preset blocked words is recorded as N. M The word segmentation embedding vector x corresponding to each preset masked word i Let be the corresponding masking word embedding vector. 1≤indexj≤N M .

[0086] The bidirectional coding network of the dense encoder in this embodiment of the invention consists of N X It consists of N bidirectional encoders connected sequentially, with a total number of encoders N. X Given a preset positive integer greater than 1; this bidirectional encoding network is used to pass through the internal N X A bidirectional encoder performs continuous bidirectional encoding on the text embedding vector X to obtain the corresponding dense text vector Y and outputs it.

[0087] Here, the text density vector Y in this embodiment of the invention is composed of N W Dense vector y for word segmentation i Composition, word segmentation dense vector y i With word segmentation embedding vector x i One-to-one correspondence; total number N M When it is not zero, it is related to the embedding vectors of each masked word. The corresponding dense word segmentation vector y i Let be the dense vector of the corresponding masking word.

[0088] It should be noted that the bidirectional encoder in this embodiment of the invention is obtained by modifying the target decoder. The modification scheme in this embodiment does not involve adjusting the model structure; it only changes the causal mask matrix M used by the target decoder in the multi-head attention layer of the mask. Causal It was replaced with an all-one matrix M all-ones .

[0089] Causal mask matrix M Causal Its function is to guide the target decoder to perform unidirectional attention inference through the setting of a semi-triangular matrix. Specifically, the decoding order of the target model is the causal mask matrix M corresponding to word-by-word decoding from left to right. Causal Given a mask matrix with all 1s in the lower left triangle and all 0s in the upper right triangle, the target decoder will only perform unidirectional attention inference based on the preceding text to the left of the current word segment; the decoding order is the causal mask matrix M corresponding to word-by-word decoding from right to left. Causal It is a mask matrix with all 1s in the lower right triangle and all 0s in the upper left triangle. At this time, the target decoder will only perform unidirectional attention reasoning based on the preceding text to the right of the current word segment.

[0090] This invention, through modification, enables the bidirectional encoder to be based on an all-one matrix M. all-ones Perform attentional reasoning because the all-one matrix M all-ones Since no zero-value mask is applied to any preceding or following text, the bidirectional encoder does not apply unidirectional masking to the preceding or following text of each word segment, thus possessing bidirectional encoding capability.

[0091] As can be seen from the foregoing, the bidirectional encoder and target decoder in this embodiment of the invention have the same model structure, both consisting of a masked multi-head attention layer, a residual connection and layer normalization unit, a feedforward neural network, and another residual connection and layer normalization unit connected sequentially, as follows: Figure 2 , 3 As shown.

[0092] The inference process of the masked multi-head attention layer of each bidirectional encoder in this embodiment of the invention is as follows:

[0093]

[0094] Where 1 ≤ index n ≤ N X ;X n-1 X n d represents the input and output vectors of the nth bidirectional encoder, respectively; n-1 For the input vector X n-1 Feature dimensions; Q n K n V n These are the query vector, key vector, and value vector corresponding to the nth bidirectional encoder, respectively. The query vector Q is respectively n Key vector K n Sum vector V n The corresponding weights; For the key vector K n The inverted vector of M; all-ones G is an all-one matrix; the parameter set G of each bidirectional encoder n By corresponding weights composition.

[0095] The only difference between the inference process of the bidirectional encoder and the target decoder in this embodiment of the invention is that the masked multi-head attention layer of the bidirectional encoder uses an all-one matrix M during inference. all-ones The target decoder's multi-head attention layer uses a causal mask matrix M during inference. Causal .

[0096] Step 2: Build a corresponding model training framework for the dense encoder; and perform a first-stage fine-tuning of the dense encoder through the masked word prediction task of the model training framework; after the first-stage fine-tuning is completed, perform a second-stage fine-tuning of the dense encoder through an unsupervised contrastive learning mechanism.

[0097] Specifically, this includes: Step 21, constructing a corresponding model training framework for the dense encoder;

[0098] Here, the model training framework of this embodiment of the invention is used to predict the original word text that is masked by a preset masking word in the masked text input to the framework and output the corresponding original word prediction sequence, such as Figure 3 As shown;

[0099] The masked text is a string without punctuation or special characters, containing at least one preset masked word but not consisting entirely of preset masked words; the original word prediction sequence consists of one or more predicted original word texts; the predicted original word texts in the original word prediction sequence correspond one-to-one with the preset masked words in the masked text; the order of all predicted original word texts in the original word prediction sequence matches the order of all preset masked words in the masked text.

[0100] like Figure 3 As shown, the model training framework of this embodiment consists of a dense encoder, a linear layer, a softmax function layer, and an output module connected sequentially; wherein:

[0101] 1) The dense encoder is used to take the masked text as the corresponding input text, perform dense vector encoding on the input text, and output the corresponding dense text vector Y to be sent to the linear layer;

[0102] 2) The linear layer is used to process the dense vectors y of each word segmentation in the dense vector Y of the text. i The word segmentation feature vector h is obtained by performing word list feature vector transformation. i ; and from all the word segmentation feature vectors h obtained i The ordered vector sequence H is sent to the Softmax function layer.

[0103] Among them, the word segmentation feature vector h i With the dense vector y of word segmentation i One-to-one correspondence; dense vectors of each blocked word The corresponding word segmentation feature vector h i The previous word segmentation feature vector h i-1 Let be the corresponding preceding word feature vector.

[0104] 3) The Softmax function layer is used to calculate the feature vectors of each preceding word. Perform vocabulary probability distribution calculations and use the results as the corresponding masked word probability vectors. And from the obtained probability vectors of all masked words The vector sequence P formed by sequential sorting is sent to the output module;

[0105] Among them, the probability vector of each masked word Each of them consists of multiple word segmentation probabilities, and the word segmentation probabilities correspond one-to-one with the word segmentation in the preset word list;

[0106] 4) The output module is used to convert the probability vectors of each blocked word. The word segmentation within the table corresponding to the maximum word segmentation probability is used as the corresponding predicted original word text; and the original word prediction sequence is formed by sorting all the obtained predicted original word texts in order and outputting it.

[0107] Step 22, and perform a one-stage fine-tuning of the dense encoder through the masked word prediction task of the model training framework;

[0108] Specifically, this includes: Step 221, randomly extracting multiple sentences from the historical text processed by the target model, and taking each extracted sentence as a corresponding first extracted text; and deleting punctuation marks and special characters in each first extracted text;

[0109] Step 222, feed the parameter set G of each bidirectional encoder in the model training framework. n The corresponding parameter set is obtained by implanting a low-rank matrix.

[0110] Here, the parameter set after implanting the low-rank matrix. By weight composition;

[0111] The masked multi-head attention layer of the bidirectional encoder is based on a parameter set. The reasoning process is as follows:

[0112]

[0113] For weight The corresponding pair of low-rank matrices, For weight The corresponding pair of low-rank matrices, For weight A corresponding pair of low-rank matrices; parameter set The corresponding low-rank matrix Form a corresponding low-rank matrix parameter set Low-rank matrix parameter sets for each bidirectional encoder This refers to the implanted additional parameter set; as will be explained below, the fine-tuning rule for each bidirectional encoder in this embodiment of the invention is: maintain the original parameter set G. n The parameter set that remains unchanged only for the implanted additional parameter set, i.e., the low-rank matrix parameter set. Modulation;

[0114] Step 223: Initialize the current occlusion rate to the preset first occlusion rate;

[0115] Where 0 < first occlusion rate < 1;

[0116] Here, the first occlusion rate in this embodiment of the invention is a preset ratio parameter, such as 15%, 20%, etc.

[0117] Step 224: Based on the word segmentation rules of the target model and according to the preset vocabulary, pre-segment each first extracted text to obtain the corresponding pre-segmented word sequence; then, use preset masking words to randomly replace the word segments of each pre-segmented word sequence according to the current masking rate to obtain the corresponding replaced word segmentation sequence; then, concatenate all the word segments of each replaced word segmentation sequence to obtain the corresponding first training text; and finally, count the total number of preset masking words in all first training texts to obtain the corresponding total number N. K ;

[0118] Among them, the sequence segmentation of the replaced word segmentation sequence contains preset masked words, and the total number of preset masked words is the integer value of the product of the total number of sequence segmentation words and the current masking rate.

[0119] For example, given a first masking rate of 20%, a preset masked word of "[MASK]", and the first extracted text of "cats love to eat sardines and pumpkins", the pre-segmented word sequence is {"cat", "love", "eat", "sardines", "and", "pumpkins"}; then, the total number of word segments in the sequence is 6, the total number of preset masked words is int(6×20%) = 1, and int() is a floor function. Taking the third word segment "sardines" in the pre-segmented word sequence as the random replacement word segment, the resulting word segmentation sequence after replacement is {"cat", "love", "eat", "[MASK]", "and", "pumpkins"}, and the first training text is "cats love to eat [MASK] and pumpkins".

[0120] Step 225: Use each of the first training texts as the corresponding masked text input to the model training framework for prediction processing, and use the probability vectors of each masked word calculated by the Softmax function layer in this processing to perform prediction processing. Let p be a corresponding prediction vector. k 1 ≤ index k ≤ N K ; and based on each prediction vector p k Set a corresponding label vector for the corresponding original word text. And by each prediction vector p k and the corresponding label vector Form a corresponding first prediction-label pair

[0121] Among them, the label vector It consists of multiple tag segmentation probabilities, and each tag segmentation probability corresponds one-to-one with the segmentation within a pre-defined vocabulary; tag vector Only one tag has a segmentation probability of 1, while the segmentation probabilities of all other tags are 0; tag vector The segmentation probability of a tag with a value of 1 corresponds to the segmentation of the original word in the table and is matched with the corresponding original word text.

[0122] Step 226, obtain N K First prediction-label pair Substitute the preset first model loss function L1 into the calculation to obtain the corresponding first loss value;

[0123] Here, the first model loss function L1 in this embodiment of the invention is:

[0124]

[0125] Step 227: Identify whether the first loss value meets the preset first loss value range; if it does, proceed to step 228; if not, identify whether the current occlusion rate is the first occlusion rate; if so, optimize all low-rank matrix parameter sets of the bidirectional coding network in the direction that minimizes the first model loss function L1, based on the preset first model optimizer. And the linear layer parameters of the linear layer are modulated once; otherwise, based on the preset second model optimizer, the optimization is performed on all low-rank matrix parameter sets in the direction that minimizes the first model loss function L1. Perform one round of modulation, and after this round of modulation is completed, return to step 225 to continue training;

[0126] Here, the first loss value range in this embodiment of the invention is a pre-set numerical range; the first and second model optimizers in this embodiment of the invention both include at least the Adam optimizer and the SGD optimizer;

[0127] It should be noted that, in the first-stage fine-tuning of this invention, two rounds of fine-tuning are performed by setting two occlusion rates (a first occlusion rate and a second occlusion rate). In this first round of fine-tuning, the bidirectional encoder network and the linear layer of the model training framework are trained synchronously, and the modulation parameter object includes all low-rank matrix parameter sets of the bidirectional encoder network. And the linear layer parameters of the linear layers; during the first round of fine-tuning, the linear layer parameters of the linear layers are no longer modulated, but only the parameter set of all low-rank matrices of the bidirectional coding network is modulated. Modulation is performed; the optimizer used in the first round of fine-tuning is the first model optimizer, and the optimizer used in the second round of fine-tuning is the second model optimizer;

[0128] Step 228: Identify whether the current occlusion rate is the first occlusion rate; if yes, reset the current occlusion rate to the preset second occlusion rate and return to step 224 to continue training after resetting; if no, stop training and confirm that the first stage of fine-tuning is over.

[0129] Where 0 < first occlusion rate < second occlusion rate < 1;

[0130] Here, the second occlusion rate in this embodiment of the invention is a preset ratio parameter, which should be greater than the first occlusion rate, such as 80%, 90%, etc.

[0131] It should be noted that, as mentioned above, the embodiments of the present invention perform two rounds of fine-tuning by setting two occlusion rates during the first-stage fine-tuning. In practical applications, more occlusion rates can be set to perform more rounds of fine-tuning. For example, it has been provided that four occlusion rates (first occlusion rate = 20%, second occlusion rate = 40%, third occlusion rate = 60%, fourth occlusion rate = 80%) are set to perform four rounds of fine-tuning. It must be noted that, regardless of whether two or more occlusion rates are set for two or more rounds of fine-tuning, the linear layer parameters of the linear layer are modulated only during the first round of fine-tuning. In the subsequent second or subsequent rounds of fine-tuning, only the bidirectional coding network is modulated, and the linear layer is no longer modulated.

[0132] Step 23: After the first-stage fine-tuning is completed, the dense encoder is fine-tuned in the second stage through an unsupervised contrastive learning mechanism.

[0133] Specifically, this includes: Step 231, randomly extracting multiple sentences from the historical text processed by the target model, and treating each extracted sentence as a corresponding second extracted text; deleting punctuation marks and special characters from each second extracted text; and counting the total number of second extracted texts to obtain the corresponding total number N. U ;

[0134] Step 232: Input each second extracted text as the corresponding input text into the dense encoder for dense vector encoding processing, and record the text dense vector Y output by this processing as the corresponding first sample;

[0135] Step 233: Each second extracted text is used as the corresponding input text and input again into the dense encoder for dense vector encoding. During this process, multiple residual connection and layer normalization units are randomly selected from the bidirectional encoding network as corresponding random scrambling units. At the end of the inference process of each random scrambling unit, a portion of the output vector of the current unit is randomly scrambled and the scrambled unit output vector is passed downstream. The dense vector Y of the text finally output by this dense vector encoding process is recorded as the corresponding second sample.

[0136] Here, the random scrambling method in this embodiment of the invention is: setting the selected sub-vector to an all-zero vector or adding Gaussian noise to the selected sub-vector;

[0137] Step 234, denote any first sample as the corresponding sample s. u 1 ≤ index u ≤ N U ; and will be compared with each sample su The other different first samples are denoted as the current sample s. u negative samples 1≤index v≤N U -1; and from each sample s u The corresponding N U -1 negative sample Form a corresponding negative sample set And with each sample s u The corresponding second sample is denoted as the current sample s. u Positive samples And by each sample s u and its corresponding positive samples With negative sample set Form a corresponding sample data group

[0138] Step 235, obtain N U Sample data sets The corresponding second loss value is obtained by substituting the preset second model loss function L2 into the calculation.

[0139] Here, the second model loss function L2 in this embodiment of the invention is:

[0140]

[0141] Where λ is a preset adjustment parameter;

[0142] sim() is the cosine similarity function for vectors. For sample s u With the corresponding positive samples cosine similarity, For sample s u With the corresponding negative samples Cosine similarity;

[0143] Step 236: Identify whether the second loss value meets the preset range of the second loss value; if not, then based on the preset third model optimizer, optimize all low-rank matrix parameter sets of the dense encoder in the direction that minimizes the second model loss function L2. Perform one round of modulation, and return to step 232 to continue training after this round of modulation is completed; if the conditions are met, stop training and confirm that the second-stage fine-tuning is complete;

[0144] Here, the second loss value range in this embodiment of the invention is a pre-set numerical range; the third model optimizer in this embodiment of the invention includes at least the Adam optimizer and the SGD optimizer.

[0145] Step 3: After the second-stage fine-tuning is completed, the user-specified document library is used as the target document library; a document vector library is constructed for the target document library using a dense encoder; and retrieval services are provided for the target document library based on the document vector library and the dense encoder.

[0146] Specifically, this includes: Step 31, after the second-stage fine-tuning is completed, the document library specified by the user is used as the target document library;

[0147] Here, the target document library of this embodiment of the invention includes a plurality of first documents;

[0148] Step 32, and use a dense encoder to build a document vector library for the target document library;

[0149] Here, the document vector library of this embodiment of the invention includes a plurality of first dense vectors, and each first dense vector corresponds one-to-one with a first document;

[0150] Specifically, this includes: taking each first document in the target document library as the corresponding input text and inputting it into a dense encoder for dense vector encoding processing, and taking the dense text vector output by the current processing as a corresponding first dense vector; and forming a corresponding document vector library from all the obtained first dense vectors;

[0151] Step 33, and provide retrieval services for the target document library based on the document vector library and dense encoder;

[0152] Specifically, this includes: using the query text entered by the user each time as the corresponding current query text; inputting the current query text as the corresponding input text into a dense encoder for dense vector encoding processing, and using the text dense vector output from the current processing as the corresponding current dense vector; calculating the cosine similarity between the current dense vector and each first dense vector in the document vector library to obtain the corresponding first similarity; recording the first dense vector whose first similarity exceeds a preset similarity threshold as the corresponding retrieval vector; and when the total number of retrieval vectors is not zero, recording the first document in the target document library corresponding to each retrieval vector as the corresponding retrieval document; and sorting all the obtained retrieval documents in descending order of first similarity to form a corresponding retrieval document sequence and feeding it back to the current user. Here, the similarity threshold in this embodiment of the invention is a preset threshold parameter.

[0153] Figure 4 This is a module structure diagram of a processing device for a dense encoder based on a large language model, provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 4As shown, the device includes: a model modification module 201, a model fine-tuning module 202, and a retrieval application module 203.

[0154] The model modification module 201 is used to select a large language model that has been pre-trained and fine-tuned for NLP tasks and implemented based on a pure decoder architecture as the corresponding target model; and to use the decoder of the target model as the corresponding target decoder; and to obtain a bidirectional encoder by modifying the causal mask matrix used by the target decoder in the inference process into an all-1 matrix; and to form a corresponding bidirectional encoding network by sequentially connecting multiple bidirectional encoders, and to form a dense encoder by connecting the embedding encoding module of the target model and the bidirectional encoding network; the NLP tasks include at least text generation tasks, information extraction tasks, and question answering tasks.

[0155] The model fine-tuning module 202 is used to build a corresponding model training framework for the dense encoder; and to perform a first-stage fine-tuning of the dense encoder through the masked word prediction task of the model training framework; after the first-stage fine-tuning is completed, a second-stage fine-tuning of the dense encoder is performed through an unsupervised contrastive learning mechanism.

[0156] The retrieval application module 203 is used to take the user-specified document library as the target document library after the two-stage fine-tuning is completed; and to construct a document vector library for the target document library using a dense encoder; and to provide retrieval services for the target document library based on the document vector library and the dense encoder; the target document library includes multiple first documents; the document vector library includes multiple first dense vectors, and the first dense vectors correspond one-to-one with the first documents.

[0157] The processing device for a dense encoder based on a large language model provided in this embodiment of the invention can execute the method steps in the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0158] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the model modification module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0159] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0160] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0161] Figure 5 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 5 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0162] exist Figure 5The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0163] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0164] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0165] This invention provides a processing method, apparatus, electronic device, and computer-readable storage medium for a dense encoder based on a large language model. As described above, this invention first selects a large language model that has completed pre-training and NLP task fine-tuning and is implemented based on a pure decoder architecture as the corresponding target model. The decoder of the target model is then used as the corresponding target decoder. A bidirectional encoder is obtained by solidifying the causal mask matrix used by the target decoder during inference into an all-one matrix. Multiple bidirectional encoders are sequentially connected to form a corresponding bidirectional encoding network. The embedding encoding module of the target model and the bidirectional encoding network are then connected to form a dense encoder. Next, a corresponding model training framework is constructed for the dense encoder, and a first-stage fine-tuning is performed using the masked word prediction task of this model training framework. After the first-stage fine-tuning, a second-stage fine-tuning is performed using an unsupervised contrastive learning mechanism. After the second-stage fine-tuning, a user-specified document library is used as the target document library. A document vector library is constructed for the target document library using the dense encoder, and a retrieval service is provided for the target document library based on the document vector library and the dense encoder. This invention modifies the decoder of the target model to obtain a dense encoder with bidirectional encoding capabilities. A first-stage fine-tuning ensures that the dense encoder can successfully inherit the ability to capture long-range dependencies, and a second-stage fine-tuning further enhances the semantic representation capabilities of the dense encoder. The dense encoder of this invention effectively improves the retrieval accuracy and recall rate for document retrieval tasks.

[0166] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0167] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A processing method for a dense encoder based on a large language model, characterized in that, The method includes: A large language model that has been pre-trained and fine-tuned for NLP tasks and implemented based on a pure decoder architecture is selected as the corresponding target model; the decoder of the target model is selected as the corresponding target decoder; a bidirectional encoder is obtained by modifying the causal mask matrix used by the target decoder during inference by fixing it into an all-1 matrix; multiple bidirectional encoders are sequentially connected to form a corresponding bidirectional encoding network, and the embedding encoding module of the target model and the bidirectional encoding network are connected to form a dense encoder; the NLP task includes at least text generation, information extraction, and question answering tasks; A corresponding model training framework is constructed for the dense encoder; and the dense encoder is fine-tuned in one stage through the masked word prediction task of the model training framework; after the first stage of fine-tuning is completed, the dense encoder is fine-tuned in two stages through an unsupervised contrastive learning mechanism. After the second-stage fine-tuning is completed, the user-specified document library is used as the target document library; and the dense encoder is used to construct a document vector library for the target document library; and retrieval services are provided for the target document library based on the document vector library and the dense encoder; the target document library includes multiple first documents; the document vector library includes multiple first dense vectors, and the first dense vectors correspond one-to-one with the first documents; The dense encoder is used to perform dense vector encoding on the input text of the encoder and output the corresponding dense text vector Y. The bidirectional encoder and the target decoder have the same model structure, both consisting of a masked multi-head attention layer, a residual connection and layer normalization unit, a feedforward neural network, and another residual connection and layer normalization unit connected in sequence. The model training framework is used to predict the original word text that is masked by a preset masking word in the masked text input to the framework and output the corresponding original word prediction sequence. The model training framework consists of the dense encoder, a linear layer, a softmax function layer, and an output module connected in sequence. The step of fine-tuning the dense encoder through the masked word prediction task of the model training framework specifically includes: Step 41: Randomly extract multiple sentences from the historical text processed by the target model, and take each extracted sentence as a corresponding first extracted text; and delete punctuation marks and special characters in each first extracted text; Step 42, feed the parameter set G of each of the bidirectional encoders of the model training framework. n The corresponding parameter set is obtained by implanting a low-rank matrix. ; Wherein, the parameter set By weight , , composition; The masked multi-head attention layer of the bidirectional encoder is based on the parameter set. The reasoning process is as follows: , ， ， ， = ， = ， = ， ， ； , For the weight The corresponding pair of low-rank matrices, , For the weight The corresponding pair of low-rank matrices, , For the weight The corresponding pair of low-rank matrices; the parameter set The corresponding low-rank matrix , , , , , Form a corresponding low-rank matrix parameter set ; Step 43: Initialize the current occlusion rate to the preset first occlusion rate; Where 0 < first occlusion rate < 1; Step 44: Based on the word segmentation rules of the target model and according to the preset vocabulary, pre-segment each of the first extracted texts to obtain the corresponding pre-segmented word sequences; and use the preset masking words to randomly replace the word segments of each of the pre-segmented word sequences according to the current masking rate to obtain the corresponding replaced word segments; and concatenate all the word segments of each of the replaced word segments to obtain the corresponding first training text; and count the total number of the preset masking words in all the first training texts to obtain the corresponding total number N. K ; Wherein, the sequence segmentation of the replaced word segmentation sequence contains the preset masked words, and the total number of the preset masked words is the integer value of the product of the total number of sequence segmentation words and the current masking rate. Step 45: Input each of the first training texts as the corresponding masked texts into the model training framework for prediction processing, and use the probability vectors of each masked word calculated by the Softmax function layer in this processing to perform prediction processing. Let p be a corresponding prediction vector. k 1 ≤ index k ≤ N K ; and based on each of the prediction vectors p k Set a corresponding label vector for the corresponding original word text. ; and by each of the predicted vectors p k and the corresponding label vector Form a corresponding first prediction-label pair (p k , ); Wherein, the label vector The tag vector consists of multiple tag segmentation probabilities, each corresponding one-to-one with the segmentation probability within the preset vocabulary; Only one of the tags has a segmentation probability of 1, while the segmentation probabilities of the rest are all 0; the tag vector The word segmentation probability of the tag with a value of 1 corresponds to the word segmentation in the table and is matched with the corresponding original word text. Step 46, obtain N K The first predicted-label pair (p) k , Substitute the first model loss function L1 into the preset value to calculate the corresponding first loss value; Wherein, the loss function L1 of the first model is: ； Step 47: Identify whether the first loss value meets the preset first loss value range; if it does, proceed to step 48; if not, identify whether the current occlusion rate is the first occlusion rate; if so, optimize all the low-rank matrix parameter sets of the bidirectional coding network in the direction that minimizes the first model loss function L1, based on the preset first model optimizer. And the linear layer parameters of the linear layer are modulated once; otherwise, based on a preset second model optimizer, the parameters of all the low-rank matrix are adjusted in the direction that minimizes the first model loss function L1. Perform one round of modulation, and after this round of modulation is completed, return to step 45 to continue training; The first and second model optimizers each include at least the Adam optimizer and the SGD optimizer; Step 48: Identify whether the current occlusion rate is the first occlusion rate; if yes, reset the current occlusion rate to the preset second occlusion rate and return to step 44 to continue training after resetting; if no, stop training and confirm that the first stage of fine-tuning is over. Where 0 < first occlusion rate < second occlusion rate < 1; The two-stage fine-tuning of the dense encoder through an unsupervised contrastive learning mechanism specifically includes: Step 51: Randomly extract multiple sentences from the historical text processed by the target model, and treat each extracted sentence as a corresponding second extracted text; delete punctuation marks and special characters in each second extracted text; and count the total number of second extracted texts to obtain the corresponding total number N. U ; Step 52: Input each of the second extracted texts as the corresponding input texts into the dense encoder for dense vector encoding processing, and record the text dense vector Y output by this processing as the corresponding first sample; Step 53: Each of the second extracted texts is used as the corresponding input text and input again into the dense encoder for dense vector encoding processing. During this processing, multiple residual connection and layer normalization units are randomly selected from the bidirectional encoding network as corresponding random scrambling units. At the end of the inference process of each random scrambling unit, a portion of the current unit's output vector is randomly scrambled, and the scrambled unit's output vector is passed downstream. The text dense vector Y finally output by this dense vector encoding processing is recorded as the corresponding second sample. The random scrambling method is as follows: setting the selected sub-vector to an all-zero vector or adding Gaussian noise to the selected sub-vector; Step 54, denote any one of the first samples as the corresponding sample s. u 1 ≤ index u ≤ N U ; and will be combined with each of the aforementioned samples s u The other different first samples are denoted as the current sample s. u negative samples 1 ≤ index v ≤ N U -1; and by each of the aforementioned samples s u The corresponding N U -1 of the negative samples Form a corresponding negative sample set { }; and will be combined with each of the aforementioned samples s u The corresponding second sample is denoted as the current sample s. u Positive samples ; and by each of the aforementioned samples s u and the corresponding positive samples With the negative sample set { } to form a corresponding sample data group (s u , ,{ }); Step 55, obtain N U The sample data groups (s) u , ,{ Substituting the preset second model loss function L2 into the input, the corresponding second loss value is obtained. Wherein, the loss function L2 of the second model is: ； λ is a preset adjustment parameter; sim() is the cosine similarity function of the vectors, sim(s u , ) is the sample s u With the corresponding positive sample cosine similarity, sim(s u , ) is the sample s u With the corresponding negative sample Cosine similarity; Step 56: Identify whether the second loss value meets the preset second loss value range; if not, then based on the preset third model optimizer, optimize all low-rank matrix parameter sets of the dense encoder in the direction that minimizes the second model loss function L2. Perform one round of modulation, and return to step 52 to continue training after this round of modulation is completed; if the conditions are met, stop training and confirm that the second-stage fine-tuning is complete; The third model optimizer includes at least the Adam optimizer and the SGD optimizer.

2. The processing method of the dense encoder based on a large language model according to claim 1, characterized in that, The dense encoder is composed of the embedded coding module and the bidirectional coding network connected together; The embedding encoding module is used to identify whether the input text contains the preset masking word "[MASK]". If it does not contain the preset masking word, the input text is segmented according to the segmentation rules of the target model and the preset vocabulary of the target model to obtain the corresponding current segmentation sequence. If it contains the preset masking word, the input text is segmented into multiple text segments without the preset masking word at each preset masking word as the cutting position, and each text segment is segmented according to the segmentation rules of the target model and the preset vocabulary to obtain the corresponding segment segmentation sequence. Each of the preset masked words is treated as an independent masked word segment, and the segmented words of all the segmented word sequences and all the masked words are sorted according to the order of arrangement in the input text to obtain the corresponding current word segment sequence; a preset starting word "[SOS]" is added to the beginning position of the current word segment sequence, and the current word segment sequence with the added starting word is taken as the corresponding text word segment sequence W; and the text word segment sequence W is embedded and encoded according to the embedding encoding rules of the target model to obtain the corresponding text embedding vector X, which is then sent to the bidirectional coding network; The text segmentation sequence W consists of multiple segments w i Sort by, 1 ≤ index i ≤ N W N W The total number of words in the text segmentation sequence; the text embedding vector X is composed of N W Each word segmentation embedding vector x i Composed of the word segmentation embedding vector x i With the word segmentation w i One-to-one correspondence; when the input text contains the preset blocked words, the total number of the preset blocked words is denoted as N. M The word segmentation embedding vector x corresponding to each of the preset masking words i Let be the corresponding masking word embedding vector. 1 ≤ index j ≤ N M ; The bidirectional coding network consists of N X The bidirectional encoders are connected sequentially, and the total number of encoders is N. X It is a preset positive integer greater than 1; the bidirectional encoding network is used to pass through the internal N X The bidirectional encoder performs continuous bidirectional encoding on the text embedding vector X to obtain the corresponding dense text vector Y and outputs it; the dense text vector Y is composed of N... W Dense vector y for word segmentation i Composed of the word segmentation dense vector y i With the word segmentation embedding vector x i One-to-one correspondence; total number N M When it is not zero, it is related to the embedding vectors of each of the aforementioned masking words. The corresponding word segmentation dense vector y i Let be the dense vector of the corresponding masking word. ; The inference process of the masked multi-head attention layer of each of the bidirectional encoders is as follows: , ， ， ， ， ； 1 ≤ index n ≤ N X ;X n-1 X n These are the input and output vectors of the nth bidirectional encoder, respectively; d n-1 For the input vector X n-1 Feature dimensions; Q n K n V n These are the query vector, key vector, and value vector corresponding to the nth bidirectional encoder; , , The query vector Q is respectively n The key vector K n and the value vector V n The corresponding weights; Let K be the key vector. n The inverted vector of M; all-ones It is an all-one matrix; the parameter set G of each of the bidirectional encoders n By corresponding weights , , composition; The only difference between the inference process of the bidirectional encoder and the target decoder is that the masked multi-head attention layer of the bidirectional encoder uses an all-one matrix M during inference. all-ones The target decoder's masked multi-head attention layer uses a causal mask matrix M during inference. Causal The decoding order of the target model is the causal mask matrix M corresponding to word-by-word decoding from left to right. Causal The mask matrix is ​​a matrix with all 1s in the lower left triangle and all 0s in the upper right triangle. The decoding order is the same as the causal mask matrix M corresponding to word-by-word decoding from right to left. Causal It is a mask matrix with all 1s in the lower right triangle and all 0s in the upper left triangle.

3. The processing method of the dense encoder based on a large language model according to claim 2, characterized in that, The blocked text is a string without punctuation marks and special characters, which contains at least one of the preset blocked words but is not entirely composed of the preset blocked words; The original word prediction sequence consists of one or more predicted original word texts; the predicted original word texts in the original word prediction sequence correspond one-to-one with the preset blocked words in the blocked text; the order of all the predicted original word texts in the original word prediction sequence matches the order of all the preset blocked words in the blocked text; The dense encoder is used to take the masked text as the corresponding input text, perform dense vector encoding on the input text, and output the corresponding dense vector Y of the text to be sent to the linear layer; The linear layer is used to process each of the word segmentation dense vectors y in the text dense vector Y. i The word segmentation feature vector h is obtained by performing word list feature vector transformation. i ; and from all the obtained word segmentation feature vectors h i The ordered sorted vector sequence H is sent to the Softmax function layer; the word segmentation feature vector h i With the word segmentation dense vector y i One-to-one correspondence; and dense vectors of each of the aforementioned masked words. The corresponding word segmentation feature vector h i The previous word segmentation feature vector h i-1 Let be the corresponding preceding word feature vector. ; The Softmax function layer is used to determine the feature vectors of each preceding word. Perform vocabulary probability distribution calculations and use the results as the corresponding masked word probability vectors. ; and the probability vectors of all the masked words obtained The sequentially ordered vector sequence P is sent to the output module; each of the masked word probability vectors Each is composed of multiple word segmentation probabilities, and the word segmentation probabilities correspond one-to-one with the word segmentation in the preset word list; The output module is used to process the probability vectors of each of the masked words. The word segmentation within the table corresponding to the maximum word segmentation probability is used as the corresponding predicted original word text; and the original word prediction sequence is formed by sequentially sorting all the obtained predicted original word texts and outputting it.

4. The processing method of the dense encoder based on a large language model according to claim 1, characterized in that, The step of constructing a document vector library for the target document library using the dense encoder specifically includes: Each of the first documents in the target document library is used as the corresponding input text and input into the dense encoder for dense vector encoding processing. The dense vector of the text output in this processing is used as a corresponding first dense vector. All the obtained first dense vectors are used to form the corresponding document vector library.

5. The processing method of the dense encoder based on a large language model according to claim 1, characterized in that, The provision of retrieval services for the target document library based on the document vector library and the dense encoder specifically includes: Each user-inputted query text is used as the current query text; the current query text is used as the input text and input into the dense encoder for dense vector encoding processing, and the text dense vector output by the current processing is used as the current dense vector; the cosine similarity between the current dense vector and each of the first dense vectors in the document vector library is calculated to obtain the corresponding first similarity; the first dense vectors whose first similarity exceeds a preset similarity threshold are recorded as the corresponding retrieval vectors; when the total number of retrieval vectors is not zero, the first documents in the target document library corresponding to each of the retrieval vectors are recorded as the corresponding retrieval documents; and all the obtained retrieval documents are sorted in descending order of the first similarity to form a corresponding retrieval document sequence and fed back to the current user.

6. An apparatus for performing the processing method of a dense encoder based on a large language model as described in any one of claims 1-5, characterized in that, The device includes: a model modification module, a model fine-tuning module, and a retrieval application module; The model modification module is used to select a large language model that has been pre-trained and fine-tuned for NLP tasks and implemented based on a pure decoder architecture as the corresponding target model; and to use the decoder of the target model as the corresponding target decoder; and to obtain a bidirectional encoder by modifying the causal mask matrix used by the target decoder during inference to a matrix of all 1s; and to form a corresponding bidirectional encoding network by sequentially connecting multiple bidirectional encoders, and to form a dense encoder by connecting the embedding encoding module of the target model and the bidirectional encoding network; the NLP task includes at least a text generation task, an information extraction task, and a question answering task; The model fine-tuning module is used to construct a corresponding model training framework for the dense encoder; and to perform a first-stage fine-tuning of the dense encoder through the masked word prediction task of the model training framework; after the first-stage fine-tuning is completed, a second-stage fine-tuning of the dense encoder is performed through an unsupervised contrastive learning mechanism. The retrieval application module is used to, after the two-stage fine-tuning is completed, take the user-specified document library as the target document library; and use the dense encoder to construct a document vector library for the target document library; and provide retrieval services for the target document library based on the document vector library and the dense encoder; the target document library includes multiple first documents; the document vector library includes multiple first dense vectors, and the first dense vectors correspond one-to-one with the first documents.

7. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-5; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-5.

Images