Document retrieval method and device with large language model as encoder

Abstract

The embodiment of the invention relates to a document retrieval method and device with a large language model as an encoder. The method comprises the steps that the large language model which is pre-trained and can process K-level contexts at a time is selected as a query, document and joint encoder; constructing a similarity scoring model by taking a query encoder and a document encoder as a core, constructing a correlation scoring model by taking a joint encoder as a core, and performing model training; and after training is finished, performing document primary selection on a document library specified by the user according to a query text input by the user and the similarity scoring model, and performing document final selection on a primary selection document sequence according to the query text input by the user and the correlation scoring model to obtain a final selection document sequence which is fed back to the current user. The retrieval accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a document retrieval method and device using a large language model as an encoder. Background Art

[0002] Traditional text retrieval systems usually rely on keyword matching or rule-based methods to handle information retrieval tasks. Through practice, we found that these conventional retrieval methods are competent for handling short text retrieval tasks; however, when dealing with document retrieval tasks, due to the large amount of information in the document and the relatively complex context semantics, these conventional retrieval methods often suffer from a problem of decreased retrieval accuracy.

[0003] Pre-trained large language models (LLMs) have strong context storage capabilities and context semantic understanding capabilities. For example, the pre-trained LLaMA series models (LLaMA-1, LLaMA-2, etc.) can process context content with a capacity of 4K (1K = 1024 words) at one time. If a pre-trained large language model (such as the LLaMA series models) can be introduced into the document retrieval task, it is bound to effectively improve the retrieval accuracy. And how to use a pre-trained large language model to handle the document retrieval task is the technical problem to be solved by the present invention. Summary of the Invention

[0004] The objective of the present invention is to provide a document retrieval method, device, electronic device, and computer-readable storage medium using a large language model as an encoder, aiming at the defects of the prior art. The present invention selects a pre-trained large language model that can process K-level context at one time as the query, document, and joint encoders; constructs a similarity scoring model with the query and document encoders as the core, and constructs a relevance scoring model with the joint encoder as the core; trains the similarity and relevance scoring models respectively; when the dataset scale is large, training is carried out in the way of fine-tuning all model parameters, and when the dataset scale is small, to avoid overfitting, low-rank matrix parameters are implanted into the three types of encoders and training is carried out by fine-tuning the low-rank matrix; after the overall training is completed, the similarity scoring model is used to perform document retrieval on the document library specified by the user according to the query text input by the user to obtain a primary selected document sequence, and the relevance scoring model is used to perform final selection of the primary selected document sequence to obtain the corresponding final selected document sequence and feedback it to the current user. The present invention introduces a pre-trained large language model into the document retrieval task, and the retrieval accuracy can be improved through the present invention.

[0005] To achieve the above objective, in the first aspect of the embodiments of the present invention, a document retrieval method using a large language model as an encoder is provided, and the method includes:

[0006] Select a large language model that has been pre-trained and can process K-level contexts at one time as the three types of encoders: query encoder, document encoder, and joint encoder; and record the maximum context capacity processed by the large language model at one time as the corresponding context capacity threshold; the context capacity threshold is in thousands of words; the large language model is preferably the LLaMA series model;

[0007] Construct a similarity scoring model with the query and document encoders as the core; and construct a relevance scoring model with the joint encoder as the core; the similarity scoring model is used to perform similarity scoring processing on the query text and document text input to the model and output the corresponding similarity score; the relevance scoring model is used to perform relevance scoring processing on the query text and the document text input to the model and output the corresponding relevance score;

[0008] Train the similarity and relevance scoring models respectively based on a preset first data set; the first data set includes multiple first data records; the first data record includes a training query text and a positive sample document set; the positive sample document set consists of multiple positive sample documents; the number of words in each positive sample document is less than or equal to the context capacity threshold; the two training query texts of any two first data records are not relevant; the training query text of each first data record is only relevant to each positive sample document of the current first data record;

[0009] After the overall model training is completed, receive the query text input by the user as the corresponding first query text; and use the currently user-specified document library as the corresponding first document library; and perform a preliminary document selection on the first document library according to the first query text, the context capacity threshold, and the similarity scoring model to obtain a preliminary document sequence; and perform a final document selection on the preliminary document sequence according to the first query text, the context capacity threshold, and the relevance scoring model to obtain a final document sequence and feedback it to the current user; the first document library includes multiple first documents; the preliminary and final document sequences are both sorted by multiple first documents.

[0010] Preferably, the first model input end of the similarity scoring model is used to receive the query text, the second model input end is used to receive the document text, and the model output end is used to output the similarity score;

[0011] The similarity scoring model includes a first preprocessing module, the query encoder, the document encoder, and a similarity calculation module; the first and second input ends of the first preprocessing module are respectively connected to the first and second model input ends of the similarity scoring model, the first output end is connected to the input end of the query encoder, and the second output end is connected to the input end of the document encoder; the output ends of the query and document encoders are respectively connected to the first and second input ends of the similarity calculation module; the output end of the similarity calculation module is connected to the model output end of the similarity scoring model;

[0012] The first preprocessing module is configured to perform word segmentation processing on the query text and the document text input to the model according to the word segmentation rules of the large language model to obtain corresponding query word segmentation sequences and document word segmentation sequences; and perform embedding encoding processing on the query and document word segmentation sequences according to the word segmentation embedding encoding rules of the large language model to obtain corresponding query embedding encoding vectors and document embedding encoding vectors; and send the query embedding encoding vector to the query encoder and send the document embedding encoding vector to the document encoder;

[0013] The query encoder is configured to perform text semantic feature extraction processing on the input query embedding encoding vector to obtain a corresponding query feature vector and send it to the similarity calculation module;

[0014] The document encoder is configured to perform text semantic feature extraction processing on the input document embedding encoding vector to obtain a corresponding document feature vector and send it to the similarity calculation module;

[0015] The similarity calculation module is configured to calculate the similarity of the query and document feature vectors according to a preset vector similarity algorithm and output the calculation result as the corresponding similarity score; the vector similarity algorithm at least includes a cosine vector similarity algorithm; the score of the similarity score is between 0 and 1.

[0016] Preferably, the first model input end of the relevance scoring model is used to receive the query text, the second model input end is used to receive the document text, and the model output end is used to output the relevance score;

[0017] The relevance scoring model includes a second preprocessing module, the joint encoder, and an MLP model; the first and second input ends of the second preprocessing module are respectively connected to the first and second model input ends of the relevance scoring model, the output end is connected to the input end of the joint encoder; the output end of the joint encoder is connected to the input end of the MLP model; the output end of the MLP model is connected to the model output end of the relevance scoring model;

[0018] The second preprocessing module is used to perform word segmentation on the query text and the document text input to the model according to the word segmentation rules of the large language model to obtain corresponding query word segmentation sequences and document word segmentation sequences; and by adding a preset separator word 'SEP' between the query word segmentation sequence and the document word segmentation sequence, sequence connection is performed on the query and document word segmentation sequences to obtain a corresponding combined word segmentation sequence = query word segmentation sequence ⊕ 'SEP' ⊕ document word segmentation sequence, where ⊕ is a string concatenation operator; and embedding encoding processing is performed on the combined word segmentation sequence according to the word segmentation embedding encoding rules of the large language model to obtain a corresponding combined embedding encoding vector and send it to the combined encoder;

[0019] The combined encoder is used to perform text semantic feature extraction processing on the input combined embedding encoding vector to obtain a corresponding combined feature vector and send it to the MLP model;

[0020] The MLP model is composed of an input processing unit and an output processing unit connected; the input processing unit is composed of one or more fully connected activation layers connected, and each of the fully connected activation layers is composed of a fully connected layer and a non-linear activation function connected; the output processing unit is composed of a fully connected layer, a softmax function and an output layer connected;

[0021] The MLP model is used to perform relevance prediction based on the combined feature vector and output the obtained prediction result as the corresponding relevance score. Specifically: the input processing unit takes the input vector feature space of the output processing unit as the corresponding target feature space and performs vector mapping on the combined feature vector in the target feature space to obtain a corresponding feature mapping vector and input it into the output processing unit; and the fully connected layer of the output processing unit performs a fully connected operation on the input feature mapping vector to obtain a fully connected vector with a feature dimension of 2, and the softmax function of the output processing unit performs binary classification probability calculation of relevant and irrelevant based on the fully connected vector to obtain corresponding relevant probability and irrelevant probability, and the output layer of the output processing unit outputs the relevant probability as the corresponding relevance score; the sum of the relevant probability and the irrelevant probability is 1; the score of the relevance score is between 0 and 1.

[0022] Preferably, the similarity and relevance scoring models are respectively trained based on a preset first data set, specifically including:

[0023] Step 41, count the total number of records in the first data set to obtain a corresponding first total;

[0024] Step 42, identifying whether the first total exceeds a preset total threshold; if the first total exceeds the total threshold, setting the corresponding first parameter adjustment mode to the first mode; if the first total does not exceed the total threshold, marking the key modules in the query, document and joint encoder respectively according to the preset key module definition rules to obtain the corresponding query key module set, document key module set and joint key module set, and implanting a set of corresponding low-rank matrix parameters into the output weight parameters of each of the key modules in the query, document and joint key module set to obtain the corresponding low-rank matrix parameter set G Q , G D and G Q+D , and set the corresponding first parameter adjustment mode to the second mode;

[0025] Wherein, each of the low-rank matrix parameters is composed of a set of corresponding low-rank matrices A and B; the original module output weight parameter of each of the key modules in the query, document and joint key module set is recorded as W0; the new module output weight parameter obtained by each of the key modules in the query, document and joint key module set after implanting the corresponding low-rank matrix parameters is recorded as W * =W0+AB;The low-rank matrix parameter set G corresponding to the query encoder Q The low-rank matrix parameter set G corresponding to the query key module set is composed of all the low-rank matrix parameters corresponding to the document encoder. D The low-rank matrix parameter set G corresponding to the document key module set is composed of all the low-rank matrix parameters corresponding to the document key module set; the low-rank matrix parameter set G corresponding to the joint encoder Q+D Composed of all the low-rank matrix parameters corresponding to the joint key module set;

[0026] Step 43: training the similarity scoring model according to the first parameter adjustment mode and the first data set;

[0027] Step 44: training the relevance scoring model according to the first parameter adjustment mode and the first data set;

[0028] Step 45 : After the model training of the similarity scoring model and the relevance scoring model are completed, confirming that the overall model training is completed.

[0029] Furthermore, the training of the similarity scoring model according to the first parameter adjustment mode and the first data set specifically includes:

[0030] Step 51: Split the first data set into two sub-data sets based on a preset first split ratio and record them as a corresponding first training set and a first evaluation set;

[0031] Among them, both the first training set and the first evaluation set are composed of multiple first data records; the ratio of the total number of records in the first training set to the first evaluation set satisfies the first segmentation ratio;

[0032] Step 52, record the total number of records in the first training set as the corresponding total N1; and count the total number of positive sample documents of each first data record in the first training set to obtain the corresponding first sample total, and calculate the sum of the N1 first sample totals as the corresponding total N2; and record each training query text in the first training set as query x i , where 1 ≤ index i ≤ N1; and record each positive sample document in the first training set as document d j , where 1 ≤ sample index j ≤ N2; and based on each query x i and each document d j set a corresponding label y i,j ; and from N1 queries x i , N2 documents d j and N1×N2 labels y i,j form N1×N2 first data groups (x i , d j , y i,j );

[0033] Among them, the value of the label y i,j is 0 or 1;

[0034] The setting rule of the label y i,j is as follows: if the query x i and the document d j belong to the same first data record, the corresponding label y i,j is set to 1; if the query x i and the document d j do not belong to the same first data record, the corresponding label y i,j is set to 0;

[0035] Step 53, use the query x i and the document d j of each first data group (x i,j , d i , y j ) as the corresponding query text and document text and input them into the similarity scoring model for processing; and cache the query and document feature vectors output by the query and document encoders of the similarity scoring model during this processing as the corresponding vectors eq i,j and vector ed i,j; and the vector eq cached this time i,j , the vector ed i,j and the corresponding label y i,j form a corresponding second data group (eq i,j , ed i,j , y i,j );

[0036] Step 54, bring the obtained N1×N2 second data groups (eq i,j , ed i,j , y i,j ) into the preset model loss function L A ; and identify the first parameter tuning mode; if the first parameter tuning mode is the first mode, then based on the preset first model optimizer, modulate the full model parameters of the query and document encoders of the similarity scoring model in the direction of minimizing the model loss function L A ; if the first parameter tuning mode is the second mode, then based on the preset second model optimizer, modulate the low-rank matrix parameter sets G A , G Q , G D corresponding to the query and document encoders in the direction of minimizing the model loss function L

[0037] Among them, the model loss function L A is implemented based on the contrastive loss function, specifically:

[0038]

[0039] f() is a preset distance metric function for calculating the vector distance between two vectors input to the function; the distance metric function includes at least the cosine distance function, Euclidean distance function, Jaccard distance function, Hamming distance function;

[0040] m is a preset distance threshold parameter;

[0041] The first model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0042] The second model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0043] Step 55, record the total number of records in the first evaluation set as the corresponding total number N3; and count the total number of positive sample documents in each of the first data records in the first evaluation set to obtain the corresponding second sample total number, and calculate the sum of the N3 second sample total numbers as the corresponding total number N4; and record each of the training query texts in the first evaluation set as the query xu , where \(1\leq u\leq N_3\); and denote each positive sample document in the first evaluation set as document \(d\) v , where \(1\leq v\leq N_4\); and based on each query \(x\) u and each document \(d\) v set a corresponding label \(y\) u,v ; and from the \(N_3\) queries \(x\) u , the \(N_4\) documents \(d\) v and the \(N_3\times N_4\) labels \(y\) u,v form \(N_3\times N_4\) third data groups \((x\) u , d v , y u,v );

[0044] Among them, the value of the label \(y\) u,v is 0 or 1;

[0045] The setting rule of the label \(y\) u,v is: if the query \(x\) u and the document \(d\) v belong to the same first data record, the corresponding label \(y\) u,v is set to 1; if the query \(x\) u and the document \(d\) v do not belong to the same first data record, the corresponding label \(y\) u,v is set to 0;

[0046] Step 56, take the query \(x\) and the document \(d\) in each third data group \((x\) u , d v , y u,v ) as the corresponding query text and document text and input them into the similarity scoring model for processing; and cache the query and document feature vectors output by the query and document encoders of the similarity scoring model during this processing and denote them as the corresponding vectors \(e_q\) u and vector \(e_d\) v ; and from the vectors \(e_q\) cached this time u,v , the vector \(e_d\) u,v and the corresponding label \(y\) u,v , the vector \(e_d\) u,v and the corresponding label \(y\) u,v form a corresponding fourth data group \((e_q\) u,v , e_d u,v , y u,v );

[0047] Step 57, for the \(N_3\times N_4\) obtained fourth data groups \((e_q\) u,v , e_d u,v , yu,v ) Substitute into the preset model evaluation function F A Perform calculations to obtain the corresponding first evaluation value;

[0048] Among them, the model evaluation function F A is:

[0049]

[0050] Step 58, identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 53 to continue training; if so, confirm that the model training of the similarity scoring model is completed.

[0051] Furthermore, the training of the correlation scoring model according to the first parameter adjustment mode and the first data set specifically includes:

[0052] Step 61, divide the first data set into two sub-data sets based on a preset second segmentation ratio, denoted as the corresponding second training set and second evaluation set;

[0053] Among them, both the second training set and the second evaluation set are composed of multiple first data records; the ratio of the total number of records in the second training set to the second evaluation set meets the second segmentation ratio;

[0054] Step 62, record the total number of records in the second training set as the corresponding total N5; and count the total number of positive sample documents of each first data record in the second training set to obtain the corresponding third sample total, and calculate the sum of the N5 third sample totals as the corresponding total N6; and denote each training query text in the second training set as query x i , 1 ≤ index i ≤ N5; and denote each positive sample document in the second training set as document d j , 1 ≤ sample index j ≤ N6; and set a corresponding label y based on the relevance between each query x i and each document d j i,j ; and form N5 × N6 fifth data groups (x i , d j , y i,j ) from N5 queries x i , N6 documents d j , and N5 × N6 labels y i,j ; i,j )

[0055] Among them, the value of the label y i,j is 0 or 1;

[0056] The label yi,j The setting rule is: if the query x i With the document d j Belong to the same first data record, then the corresponding label y i,j Set to 1; if the query x i With the document d j do not belong to the same first data record, then the corresponding label y i,j Set to 0;

[0057] Step 63: each of the fifth data groups (x i ,d j ,y i,j ) of the query x i and the document d j The corresponding query text and document text are input into the relevance scoring model for processing; and the relevant probability and irrelevant probability output by the softmax function of the MLP model within the relevance scoring model during this processing are cached and recorded as the corresponding probability pp i,j and probability pn i,j ; and the probability pp of this cache i,j , the probability pn i,j and the corresponding label y i,j Form a corresponding sixth data group (pp i,j ,pn i,j ,y i,j );

[0058] Step 64: the obtained N5×N6 sixth data groups (pp i,j ,pn i,j ,y i,j ) Bring in the preset model loss function L B ; and identify the first parameter adjustment mode; if the first parameter adjustment mode is the first mode, then based on the preset third model optimizer towards making the model loss function L B In the direction of reaching the minimum value, a round of modulation is performed on the joint encoder of the correlation scoring model and the full model parameters of the MLP model; if the first parameter adjustment mode is the second mode, the fourth model optimizer is based on the preset mode to make the model loss function L B The direction in which the minimum value is reached is the low rank matrix parameter set G corresponding to the joint encoder. Q+D and performing a round of modulation on the model parameters of the MLP model;

[0059] Among them, the model loss function L B It is implemented based on the cross entropy loss function, specifically:

[0060]

[0061] The third model optimizer includes at least an Adam optimizer and an SGD optimizer;

[0062] The fourth model optimizer includes at least an Adam optimizer and an SGD optimizer;

[0063] Step 65: record the total number of records in the second evaluation set as the corresponding total number N7; and calculate the total number of the positive sample documents of each of the first data records in the second evaluation set to obtain the corresponding fourth sample total number, and calculate the sum of the N7 fourth sample total numbers as the corresponding total number N8; and record each of the training query texts in the second evaluation set as query x u , 1≤index u≤N7; and each of the positive sample documents in the second evaluation set is recorded as document d v , 1≤sample index v≤N8; and based on each of the query x u With each of the said documents d v The correlation sets a corresponding label y u,v ; and by N7 said queries x u , N8 of the said documents d v and N7×N8 labels y u,v Composed of N7×N8 seventh data groups (x u ,d v ,y u,v );

[0064] Among them, the label y u,v The value of is 0 or 1;

[0065] The label y u,v The setting rule is: if the query x u With the document d v Belong to the same first data record, then the corresponding label y u,v Set to 1; if the query x u With the document d v do not belong to the same first data record, then the corresponding label y u,v Set to 0;

[0066] Step 66: each of the seventh data groups (x u ,d v ,y u,v ) of the query x u and the document d vThe corresponding query text and the document text are input into the relevance scoring model for processing, and the relevance score output by the model this time is recorded as the corresponding predicted score y * u,v ; and from the predicted score y obtained this time * u,v and the corresponding label y i,j to form a corresponding eighth data group (y * u,v , y i,j );

[0067] Step 67, bring the obtained N7×N8 eighth data groups (y * u,v , y i,j ) into the preset model evaluation function F B for calculation to obtain the corresponding second evaluation value;

[0068] Among them, the model evaluation function F B is implemented based on the MAE function, the MSE function or the RMSE function;

[0069] When the model evaluation function F B is based on the MAE function, specifically:

[0070]

[0071] When the model evaluation function F B is based on the MSE function, specifically:

[0072]

[0073] When the model evaluation function F B is based on the RMSE function, specifically:

[0074]

[0075] Step 68, identify whether the second evaluation value meets the preset second evaluation value range; if not, return to Step 63 to continue training; if so, confirm that the model training of the relevance scoring model ends.

[0076] Preferably, the initial document selection of the first document library according to the first query text, the context capacity threshold and the similarity scoring model to obtain the initial document sequence specifically includes:

[0077] Take each of the first documents in the first document library as the corresponding current document; identify whether the number of words in the current document exceeds the context capacity threshold; if not, input the first query text and the current document as the corresponding query text and document text into the similarity scoring model for similarity scoring processing, and take the similarity score obtained this time as the corresponding first document score; if it has exceeded, perform fragmentation processing on the current document with the context capacity threshold as the maximum fragmentation word count to obtain the corresponding multiple first document fragments, and input the first query text and each of the first document fragments as the corresponding query text and document text into the similarity scoring model for similarity scoring processing, and take the similarity score obtained this time as the corresponding first fragment score, and take the maximum score among all the first fragment scores corresponding to the current document as the corresponding first document score;

[0078] Sort all the first documents in the first document library in descending order of the first document score to obtain the corresponding first document sequence; and extract the first R first documents in the first document sequence to form the corresponding initial selection document sequence; R is a preset positive integer value.

[0079] Preferably, the final selection of documents from the initial selection document sequence according to the first query text, the context capacity threshold, and the relevance scoring model to obtain the final selection document sequence and feedback to the current user specifically includes:

[0080] Take each of the first documents in the initial selection document sequence as the corresponding current document; identify whether the number of words in the current document exceeds the context capacity threshold; if not, input the first query text and the current document as the corresponding query text and document text into the relevance scoring model for relevance scoring processing, and take the relevance score obtained this time as the corresponding second document score; if it has exceeded, perform fragmentation processing on the current document with the context capacity threshold as the maximum fragmentation word count to obtain the corresponding multiple second document fragments, and input the first query text and each of the second document fragments as the corresponding query text and document text into the relevance scoring model for relevance scoring processing, and take the relevance score obtained this time as the corresponding second fragment score, and take the maximum score among all the second fragment scores corresponding to the current document as the corresponding second document score;

[0081] Sort all the first documents in the preliminary selected document sequence in descending order according to the second document score to obtain a corresponding second document sequence; remove the first documents in the second document sequence whose second document scores are lower than a preset relevance score threshold; and feedback the latest second document sequence to the current user as the corresponding final selected document sequence.

[0082] In the second aspect of the embodiments of the present invention, there is provided an apparatus for implementing the document retrieval method with a large language model as an encoder described in the first aspect above. The apparatus includes: a preprocessing module, a model construction module, a model training module, and a document retrieval module;

[0083] The preprocessing module is used to select a large language model that has completed pre-training and can process K-level contexts at one time as three types of encoders: a query encoder, a document encoder, and a joint encoder; and record the maximum context capacity processed by the large language model at one time as the corresponding context capacity threshold; the context capacity threshold is in thousands of words; the large language model preferably is the LLaMA series model;

[0084] The model construction module is used to construct a similarity scoring model with the query and document encoders as the core; and construct a relevance scoring model with the joint encoder as the core; the similarity scoring model is used to perform similarity scoring processing on the query text and the document text input to the model and output the corresponding similarity score; the relevance scoring model is used to perform relevance scoring processing on the query text and the document text input to the model and output the corresponding relevance score;

[0085] The model training module is used to train the similarity and relevance scoring models respectively based on a preset first data set; the first data set includes multiple first data records; the first data record includes a training query text and a positive sample document set; the positive sample document set consists of multiple positive sample documents; the number of words in each positive sample document is less than or equal to the context capacity threshold; the two training query texts of any two first data records are not relevant; the training query text of each first data record is only relevant to each positive sample document of the current first data record;

[0086] The document retrieval module is used to, after the overall model training is completed, receive the query text input by the user as the corresponding first query text; and use the document library specified by the current user as the corresponding first document library; and perform primary document selection on the first document library according to the first query text, the context capacity threshold, and the similarity scoring model to obtain a primary document sequence; and perform final document selection on the primary document sequence according to the first query text, the context capacity threshold, and the relevance scoring model to obtain a final document sequence and feedback it to the current user; the first document library includes a plurality of first documents; both the primary and final document sequences are sorted by a plurality of the first documents.

[0087] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0088] The processor is used to be coupled with the memory, read and execute the instructions in the memory to implement the method steps described in the first aspect above;

[0089] The transceiver is coupled with the processor, and the processor controls the transceiver to perform message sending and receiving.

[0090] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a computer, the computer is caused to execute the instructions of the method described in the first aspect above.

[0091] An embodiment of the present invention provides a document retrieval method, device, electronic device, and computer-readable storage medium using a large language model as an encoder. As can be seen from the above content, an embodiment of the present invention selects a large language model that has completed pre-training and can process K-level contexts at one time as the query, document, and joint encoder; constructs a similarity scoring model with the query and document encoders as the core, and constructs a relevance scoring model with the joint encoder as the core; and trains the similarity and relevance scoring models respectively; when the dataset scale is large, train in the way of fine-tuning all model parameters, and when the dataset scale is small, to avoid overfitting, implant low-rank matrix parameters into the three types of encoders and train by fine-tuning the low-rank matrix; after the overall training is completed, use the similarity scoring model to perform document retrieval on the document library specified by the user according to the query text input by the user to obtain a primary document sequence, and use the relevance scoring model to perform final document selection on the primary document sequence to obtain the corresponding final document sequence and feedback it to the current user. An embodiment of the present invention effectively improves the accuracy of the document retrieval task by introducing a pre-trained large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1Schematic diagram of a document retrieval method using a large language model as an encoder provided in Embodiment 1 of the present invention;

[0093] Figure 2 Module schematic diagram of the similarity scoring model and the relevance scoring model provided in Embodiment 1 of the present invention;

[0094] Figure 3 Module structure diagram of a document retrieval device using a large language model as an encoder provided in Embodiment 2 of the present invention;

[0095] Figure 4 Schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed implementation manners

[0096] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0097] Embodiment 1 of the present invention provides a document retrieval method using a large language model as an encoder, as Figure 1 shown in the schematic diagram of a document retrieval method using a large language model as an encoder provided in Embodiment 1 of the present invention. The method mainly includes the following steps:

[0098] Step 1, select a large language model that has completed pre-training and can process K-level contexts at one time as three types of encoders: a query encoder, a document encoder, and a joint encoder; and record the maximum context capacity processed by the large language model at one time as the corresponding context capacity threshold.

[0099] Here, the large language model of the embodiment of the present invention is preferably selected from the LLaMA series of models. The context capacity threshold of the large language model in the embodiment of the present invention is in units of one thousand words and should be greater than 1000 words. The large language model mentioned below is this selected model.

[0100] Step 2, construct a similarity scoring model with the query and document encoders as the core; and construct a relevance scoring model with the joint encoder as the core.

[0101] Here, the similarity scoring model and the relevance scoring model of the embodiment of the present invention, as Figure 2 shown in the module schematic diagram of the similarity scoring model and the relevance scoring model provided in Embodiment 1 of the present invention.

[0102] The similarity scoring model of the embodiments of the present invention is used to perform similarity scoring processing on the query text and document text input to the model and output the corresponding similarity score, and the score of the similarity score is between 0 and 1. The relevance scoring model of the embodiments of the present invention is used to perform relevance scoring processing on the query text and document text input to the model and output the corresponding relevance score, and the score of the relevance score is between 0 and 1.

[0103] As Figure 2 shown, the first model input end of the similarity scoring model is used to receive the query text, the second model input end is used to receive the document text, and the model output end is used to output the similarity score. Its internal model components include: a first preprocessing module, a query encoder, a document encoder, and a similarity calculation module.

[0104] The connection relationship of the internal components of the similarity scoring model is: the first and second input ends of the first preprocessing module are respectively connected to the first and second model input ends of the similarity scoring model, the first output end is connected to the input end of the query encoder, and the second output end is connected to the input end of the document encoder; the output ends of the query and document encoders are respectively connected to the first and second input ends of the similarity calculation module; the output end of the similarity calculation module is connected to the model output end of the similarity scoring model.

[0105] The component functions of the internal components of the similarity scoring model are: [[ID=?]]

[0106] 1) First preprocessing module:

[0107] The first preprocessing module of the embodiments of the present invention is used to perform word segmentation processing on the query text and document text input to the model according to the word segmentation rules of the large language model to obtain the corresponding query word segmentation sequence and document word segmentation sequence; and perform embedding encoding processing on the query and document word segmentation sequences according to the word segmentation embedding encoding rules of the large language model to obtain the corresponding query embedding encoding vector and document embedding encoding vector; and send the query embedding encoding vector to the query encoder and send the document embedding encoding vector to the document encoder;

[0108] 2) Query encoder:

[0109] As described in step 1, the query encoder of the embodiments of the present invention is implemented based on a large language model that has been pre-trained and can process K-level contexts at one time; the query encoder is used to perform text semantic feature extraction processing on the input query embedding encoding vector to obtain the corresponding query feature vector and send it to the similarity calculation module;

[0110] 3) Document encoder:

[0111] It should be noted that there seems to be an error in the original text where the "ID=?" is not properly formatted. It should be corrected to a valid ID number for a proper translation.As described in step 1, the document encoder in the embodiment of the present invention is implemented based on a large language model that is the same as the query encoder; the document encoder is used to perform text semantic feature extraction processing on the input document embedding encoding vector to obtain the corresponding document feature vector and send it to the similarity calculation module;

[0112] 4) Similarity calculation module:

[0113] The similarity calculation module in the embodiment of the present invention is used to calculate the similarity between the query and the document feature vector according to a preset vector similarity algorithm and output the calculation result as the corresponding similarity score; the vector similarity algorithm mentioned here at least includes the cosine vector similarity algorithm.

[0114] As Figure 2 shown, the first model input end of the relevance scoring model is used to receive the query text, the second model input end is used to receive the document text, and the model output end is used to output the relevance score. Its internal model components include: a second preprocessing module, a joint encoder, and an MLP model.

[0115] The connection relationship of each internal component of the relevance scoring model is: the first and second input ends of the second preprocessing module are respectively connected to the first and second model input ends of the relevance scoring model, and the output end is connected to the input end of the joint encoder; the output end of the joint encoder is connected to the input end of the MLP model; the output end of the MLP model is connected to the model output end of the relevance scoring model.

[0116] The component functions of each internal component of the relevance scoring model are:

[0117] 1) Second preprocessing module:

[0118] The second preprocessing module in the embodiment of the present invention is used to perform word segmentation processing on the query text and the document text input to the model according to the word segmentation rules of the large language model to obtain the corresponding query word segmentation sequence and document word segmentation sequence; and by adding a preset separator token 'SEP' between the query word segmentation sequence and the document word segmentation sequence, the query and document word segmentation sequences are concatenated to obtain the corresponding joint word segmentation sequence = query word segmentation sequence ⊕ 'SEP' ⊕ document word segmentation sequence, where ⊕ is the string concatenation operator; and the joint word segmentation sequence is embedded and encoded according to the word segmentation embedding encoding rules of the large language model to obtain the corresponding joint embedded encoding vector and send it to the joint encoder;

[0119] 2) Joint encoder:

[0120] As described in step 1, the joint encoder of the embodiment of the present invention is implemented based on a large language model that is the same as the query and document encoders; this joint encoder is used to perform text semantic feature extraction processing on the input joint embedding coding vector to obtain the corresponding joint feature vector and send it to the MLP model;

[0121] 3) MLP model:

[0122] The MLP model of the embodiment of the present invention is formed by connecting an input processing unit and an output processing unit; where: the input processing unit is formed by connecting one or more fully connected activation layers, and each fully connected activation layer is formed by connecting a fully connected layer and a non-linear activation function; the output processing unit is formed by connecting a fully connected layer, a softmax function, and an output layer;

[0123] This MLP model is used to perform relevance prediction based on the joint feature vector and output the obtained prediction result as the corresponding relevance score. Specifically: the input processing unit takes the input vector feature space of the output processing unit as the corresponding target feature space and performs vector mapping on the joint feature vector in the target feature space to obtain the corresponding feature mapping vector and input it into the output processing unit; and the fully connected layer of the output processing unit performs a fully connected operation on the input feature mapping vector to obtain a fully connected vector with a feature dimension of 2, and the softmax function of the output processing unit performs binary classification probability calculation of relevant and irrelevant based on the fully connected vector to obtain the corresponding relevant probability and irrelevant probability, and the output layer of the output processing unit outputs the relevant probability as the corresponding relevance score; where the sum of the relevant and irrelevant probabilities output by the softmax function should be 1.

[0124] Step 3, train the similarity and relevance scoring models respectively based on a preset first data set;

[0125] Here, the first data set of the embodiment of the present invention is a data set that is preset and used to train the similarity and relevance scoring models; this first data set includes multiple first data records; each first data record is composed of a training query text and a set of positive sample documents; each set of positive sample documents is composed of multiple positive sample documents; the number of words in each positive sample document is less than or equal to the context capacity threshold; the two training query texts of any two first data records are not relevant; the training query text of each first data record is only relevant to the positive sample documents of the current first data record and is irrelevant to the positive sample documents of any other first data record;

[0126] Specifically, it includes: Step 31, count the total number of records in the first data set to obtain the corresponding first total;

[0127] Step 32: Identify whether the first total exceeds a preset total threshold. If the first total has exceeded the total threshold, set the corresponding first tuning parameter mode to the first mode. If the first total has not exceeded the total threshold, perform key module marking on the query, document, and joint encoders respectively according to the preset key module definition rules to obtain the corresponding query key module set, document key module set, and joint key module set, and implant a set of corresponding low-rank matrix parameters on the output weight parameters of each key module in the query, document, and joint key module sets to obtain the corresponding low-rank matrix parameter set G Q 、G D and G Q+D ,and set the corresponding first tuning parameter mode to the second mode;

[0128] Here, the total threshold is a preset threshold parameter. The first total exceeding the total threshold indicates that the dataset size is large. In this case, the embodiments of the present invention perform training in the manner of fine-tuning all model parameters, and the first tuning parameter mode = the first mode. The first total not exceeding the total threshold indicates that the dataset size is small. In this case, to avoid overfitting, the embodiments of the present invention implant low-rank matrix parameters into the three types of encoders (query encoder, document encoder, joint encoder) and perform training by fine-tuning the low-rank matrices, and the first tuning parameter mode = the second mode;

[0129] It should be noted that each low-rank matrix parameter in the embodiments of the present invention is composed of a corresponding set of low-rank matrices A and B. The original module output weight parameters of each key module in the query, document, and joint key module sets are denoted as W0. The new module output weight parameters obtained after implanting the corresponding low-rank matrix parameters for each key module in the query, document, and joint key module sets are denoted as W * = W0 + AB; The low-rank matrix parameter set G Q corresponding to the query encoder is composed of all low-rank matrix parameters corresponding to the query key module set; The low-rank matrix parameter set G D corresponding to the document encoder is composed of all low-rank matrix parameters corresponding to the document key module set; The low-rank matrix parameter set G Q+D corresponding to the joint encoder is composed of all low-rank matrix parameters corresponding to the joint key module set;

[0130] Step 33: Train the similarity scoring model according to the first tuning parameter mode and the first dataset;

[0131] Specifically, it includes: Step 33-1: Divide the first dataset into two sub-datasets according to a preset first segmentation ratio and denote them as the corresponding first training set and first evaluation set;

[0132] Here, the first splitting ratio of the embodiment of the present invention is a preset ratio parameter, such as 8:2; both the first training set and the first evaluation set are composed of a plurality of first data records; the ratio of the total number of records in the first training set to the first evaluation set satisfies the first splitting ratio;

[0133] Step 332, record the total number of records in the first training set as the corresponding total number N1; and count the total number of positive sample documents of each first data record in the first training set to obtain the corresponding first sample total number, and take the calculation result of the sum of N1 first sample total numbers as the corresponding total number N2; and record each training query text in the first training set as query x i , 1 ≤ index i ≤ N1; and record each positive sample document in the first training set as document d j , 1 ≤ sample index j ≤ N2; and based on each query x i and each document d j set a corresponding label y i,j ; and from N1 queries x i , N2 documents d j and N1×N2 labels y i,j form N1×N2 first data groups (x i , d j , y i,j );

[0134] Here, the label y i,j of the embodiment of the present invention takes values of 0 or 1;

[0135] The setting rule of this label y i,j is: if the query x i and the document d j belong to the same first data record, the corresponding label y i,j is set to 1; if the query x i and the document d j do not belong to the same first data record, the corresponding label y i,j is set to 0;

[0136] Step 333, use the query x i and the document d j of each first data group (x i,j , d i , y j ) as the corresponding query text and document text and input them into the similarity scoring model for processing; and cache the query and document feature vectors output by the query and document encoders of the similarity scoring model during this processing as the corresponding vectors eq i,j and vector ed i,j ; and from the vectors eq i,j and vector ed cached this time,i,j and the corresponding label y i,j to form a corresponding second data set (eq i,j , ed i,j , y i,j );

[0137] Step 334, bring the obtained N1×N2 second data sets (eq i,j , ed i,j , y i,j ) into the preset model loss function L A ; and identify the first tuning mode; if the first tuning mode is the first mode, then based on the preset first model optimizer, towards the direction that makes the model loss function L A reach the minimum value, perform one round of modulation on the query of the similarity scoring model and all the model parameters of the document encoder; if the first tuning mode is the second mode, then based on the preset second model optimizer, towards the direction that makes the model loss function L A reach the minimum value, perform one round of modulation on the low-rank matrix parameter sets G Q 、G D corresponding to the query and the document encoder;

[0138] Here, the model loss function L of the embodiment of the present invention A is implemented based on the contrastive loss function, specifically:

[0139]

[0140] where f() is a preset distance metric function for calculating the vector distance between two vectors input to the function; the distance metric function includes at least the cosine distance function, the Euclidean distance function, the Jaccard distance function, and the Hamming distance function;

[0141] m is a preset distance threshold parameter;

[0142] The first model optimizer of the embodiment of the present invention includes at least the Adam optimizer and the SGD optimizer; the second model optimizer of the embodiment of the present invention includes at least the Adam optimizer and the SGD optimizer;

[0143] Step 335, record the total number of records in the first evaluation set as the corresponding total N3; and count the total number of positive sample documents in each first data record of the first evaluation set to obtain the corresponding second sample total, and calculate the sum of the N3 second sample totals as the corresponding total N4; and record each training query text in the first evaluation set as query x u , 1≤index u≤N3; and record each positive sample document in the first evaluation set as document d v , 1≤sample index v≤N4; and based on each query xu Set a corresponding label y for the relevance with each document d v ; and form N3×N4 third data groups (x u,v , N3 queries x u , N4 documents d v and N3×N4 labels y u,v ); u , d v , y u,v )

[0144] Here, the label y in the embodiment of the present invention u,v is similar to the label y in the previous text i,j , and its value is also 0 or 1; the setting rule is also: if the query x u and the document d v belong to the same first data record, the corresponding label y u,v is set to 1, if the query x u and the document d v do not belong to the same first data record, the corresponding label y u,v is set to 0;

[0145] Step 336, take the query x and the document d of each third data group (x u , d v , y u,v ) as the corresponding query text and document text and input them into the similarity scoring model for processing; and cache the query and document feature vectors output by the query and document encoders of the similarity scoring model during this processing, denoted as the corresponding vectors eq u and vector ed v ; and form a corresponding fourth data group (eq u,v , ed u,v , y by the vectors eq u,v and vector ed u,v cached this time and the corresponding label y u,v ); u,v , ed u,v , y u,v )

[0146] Step 337, substitute the obtained N3×N4 fourth data groups (eq u,v , ed u,v , y u,v ) into the preset model evaluation function F A for calculation to obtain the corresponding first evaluation value;

[0147] Here, the model evaluation function F in the embodiment of the present invention A is actually a calculation function of the average loss, specifically:

[0148]

[0149] Step 338: Identify whether the first evaluation value meets a preset first evaluation value range; if not, return to Step 333 to continue training; if so, confirm that the model training of the similarity scoring model is completed;

[0150] Here, the first evaluation value range in the embodiment of the present invention is a preset numerical range;

[0151] Step 34: Train the relevance scoring model according to the first parameter adjustment mode and the first data set;

[0152] Specifically, it includes: Step 341: Divide the first data set into two sub-data sets, denoted as the corresponding second training set and second evaluation set, based on a preset second segmentation ratio;

[0153] Here, the second segmentation ratio in the embodiment of the present invention is a preset ratio parameter, such as 8:2; both the second training set and the second evaluation set are composed of multiple first data records; the ratio of the total number of records in the second training set to the second evaluation set meets the second segmentation ratio;

[0154] Step 342: Denote the total number of records in the second training set as the corresponding total N5; and count the total number of positive sample documents of each first data record in the second training set to obtain the corresponding third sample total, and calculate the sum of the N5 third sample totals as the corresponding total N6; and denote each training query text in the second training set as query x i , 1 ≤ index i ≤ N5; and denote each positive sample document in the second training set as document d j , 1 ≤ sample index j ≤ N6; and set a corresponding label y based on the relevance between each query x i and each document d j ; and form N5 × N6 fifth data groups (x i,j , d i , y j ) consisting of N5 queries x i,j , N6 documents d i , and N5 × N6 labels y j , y i,j );

[0155] Among them, the value of the label y i,j is 0 or 1;

[0156] The setting rule of the label y i,j is: if the query x i and the document d j belong to the same first data record, the corresponding label y i,jSet it to 1; if querying x i and the document d j do not belong to the same first data record, then the corresponding label y i,j is set to 0;

[0157] Step 343, input the query x i , d j , y i,j ) of each fifth data group into the relevance scoring model as the corresponding query text and document text for processing; and cache the relevant probability and irrelevant probability output by the softmax function of the MLP model inside the relevance scoring model during this processing, denoted as the corresponding probability pp i and probability pn j ; and form a corresponding sixth data group (pp i,j , pn i,j , y i,j ) from the cached probability pp i,j , probability pn i,j and the corresponding label y i,j , pn i,j , y i,j );

[0158] Step 344, input the obtained N5×N6 sixth data groups (pp i,j , pn i,j , y i,j ) into the preset model loss function L B ; and identify the first tuning mode; if the first tuning mode is the first mode, then based on the preset third model optimizer, modulate the full model parameters of the joint encoder and MLP model of the relevance scoring model in the direction of minimizing the model loss function L B ; if the first tuning mode is the second mode, then based on the preset fourth model optimizer, modulate the low-rank matrix parameter set G B corresponding to the joint encoder and the model parameters of the MLP model in the direction of minimizing the model loss function L Q+D for one round;

[0159] Here, the model loss function L of the embodiment of the present invention B is implemented based on the cross-entropy loss function, specifically:

[0160]

[0161] The third model optimizer of the embodiment of the present invention at least includes the Adam optimizer and the SGD optimizer; the fourth model optimizer of the embodiment of the present invention at least includes the Adam optimizer and the SGD optimizer;

[0162] Step 345: record the total number of records in the second evaluation set as the corresponding total number N7; and calculate the total number of positive sample documents of each first data record in the second evaluation set to obtain the corresponding fourth sample total number, and calculate the sum of the N7 fourth sample total numbers as the corresponding total number N8; and record each training query text in the second evaluation set as query x u , 1≤index u≤N7; and each positive sample document in the second evaluation set is recorded as document d v , 1≤sample index v≤N8; and based on each query x u With each document d v The correlation sets a corresponding label y u,v ; and by N7 queries x u 、N8 documents v and N7×N8 labels y u,v Composed of N7×N8 seventh data groups (x u ,d v ,y u,v );

[0163] Among them, the label y u,v The value of is 0 or 1;

[0164] Tag y u,v The setting rule is: if the query x u With document d v Belong to the same first data record, the corresponding label y u,v Set to 1; if query x u With document d v Do not belong to the same first data record, then the corresponding label y u,v Set to 0;

[0165] Step 346: each seventh data group (x u ,d v ,y u,v ) query x u and document d v The corresponding query text and document text are input into the relevance scoring model for processing and the relevance score output by this model is recorded as the corresponding prediction score y u * ,v ; And the prediction score y obtained this time * u,v and the corresponding label y i,j Form a corresponding eighth data group (y * u,v ,y i,j );

[0166] Step 347: The obtained N7×N8 eighth data groups (y* u,v , y i,j ) Substitute into the preset model evaluation function F B Perform calculations to obtain the corresponding second evaluation value;

[0167] Here, the model evaluation function F of the embodiment of the present invention B Is implemented based on the MAE function, MSE function or RMSE function;

[0168] If the model evaluation function F B Is based on the MAE function, then specifically:

[0169]

[0170] If the model evaluation function F B Is based on the MSE function, then specifically:

[0171]

[0172] If the model evaluation function F B Is based on the RMSE function, then specifically:

[0173]

[0174] Step 348, Identify whether the second evaluation value meets the preset second evaluation value range; if not, return to step 343 to continue training; if so, confirm that the model training of the correlation scoring model is completed;

[0175] Here, the second evaluation value range of the embodiment of the present invention is a preset numerical range;

[0176] Step 35, After the model training of both the similarity scoring model and the correlation scoring model is completed, confirm that the overall model training is completed.

[0177] Step 4, After the overall model training is completed, receive the query text input by the user as the corresponding first query text; and use the document library specified by the current user as the corresponding first document library; and perform document primary selection on the first document library according to the first query text, context capacity threshold and similarity scoring model to obtain a primary selected document sequence; and perform document final selection on the primary selected document sequence according to the first query text, context capacity threshold and correlation scoring model to obtain a final selected document sequence and feedback it to the current user;

[0178] Among them, the first document library includes multiple first documents; both the primary selected and final selected document sequences are sorted by multiple first documents;

[0179] Specifically, it includes: Step 41, after the overall model training is completed, receive the query text input by the user as the corresponding first query text; and use the document library specified by the current user as the corresponding first document library;

[0180] Step 42, and perform initial document selection on the first document library according to the first query text, context capacity threshold, and similarity scoring model to obtain an initial document sequence;

[0181] Specifically, it includes: Step 421, use each first document in the first document library as the corresponding current document; and identify whether the number of words in the current document exceeds the context capacity threshold; if not, use the first query text and the current document as the corresponding query text and document text to input into the similarity scoring model for similarity scoring processing, and use the similarity score obtained this time as the corresponding first document score; if it has exceeded, perform fragmentation processing on the current document with the context capacity threshold as the maximum fragmentation word count to obtain the corresponding multiple first document fragments, and use the first query text and each first document fragment as the corresponding query text and document text to input into the similarity scoring model for similarity scoring processing, and use the similarity score obtained this time as the corresponding first fragment score, and use the maximum score among all the first fragment scores corresponding to the current document as the corresponding first document score;

[0182] Step 422, sort all the first documents in the first document library in descending order according to the first document score to obtain the corresponding first document sequence; and extract the first R first documents in the first document sequence to form the corresponding initial document sequence;

[0183] Here, R in the current Step 422 is a preset positive integer value;

[0184] Step 43, and perform final document selection on the initial document sequence according to the first query text, context capacity threshold, and relevance scoring model to obtain a final document sequence and feedback it to the current user;

[0185] Specifically, it includes: Step 431, taking each first document in the initially selected document sequence as the corresponding current document; and identifying whether the number of words in the current document exceeds the context capacity threshold. If it does not exceed, taking the first query text and the current document as the corresponding query text and document text and inputting them into the relevance scoring model for relevance scoring processing, and taking the relevance score obtained this time as the corresponding second document score. If it has exceeded, performing fragmentation processing on the current document with the context capacity threshold as the maximum fragmentation word count to obtain the corresponding multiple second document fragments, and taking the first query text and each second document fragment as the corresponding query text and document text and inputting them into the relevance scoring model for relevance scoring processing, and taking the relevance score obtained this time as the corresponding second fragment score, and taking the maximum score among all the second fragment scores corresponding to the current document as the corresponding second document score;

[0186] Step 432, sorting all the first documents in the initially selected document sequence in descending order according to the second document scores to obtain the corresponding second document sequence; removing the first documents in the second document sequence whose second document scores are lower than the preset relevance score threshold; and feeding back the latest second document sequence to the current user as the corresponding final selected document sequence.

[0187] Figure 3 FIG. is a module structure diagram of a document retrieval device with a large language model as the encoder provided in the second embodiment of the present invention. The device is a terminal device or a server for implementing the foregoing method embodiment, or may be a device that enables the foregoing terminal device or server to implement the foregoing method embodiment. For example, the device may be a device or a chip system of the foregoing terminal device or server. As Figure 3 shown, the device includes: a preprocessing module 201, a model construction module 202, a model training module 203, and a document retrieval module 204.

[0188] The preprocessing module 201 is used to select a large language model that has completed pre-training and can process K-level contexts at one time as three types of encoders: a query encoder, a document encoder, and a joint encoder; and record the maximum context capacity processed by the large language model at one time as the corresponding context capacity threshold; the context capacity threshold is in units of one thousand words; the large language model preferably selects the LLaMA series of models.

[0189] The model construction module 202 is used to construct a similarity scoring model with the query and document encoders as the core; and construct a relevance scoring model with the joint encoder as the core; the similarity scoring model is used to perform similarity scoring processing on the query text and the document text input into the model and output the corresponding similarity score; the relevance scoring model is used to perform relevance scoring processing on the query text and the document text input into the model and output the corresponding relevance score.

[0190] The model training module 203 is used to train the similarity and relevance scoring models respectively based on a preset first data set; the first data set includes a plurality of first data records; the first data records include training query texts and positive sample document sets; the positive sample document sets are composed of a plurality of positive sample documents; the number of words in each positive sample document is less than or equal to the context capacity threshold; the two training query texts of any two first data records are not relevant; the training query text of each first data record is only relevant to each positive sample document of the current first data record.

[0191] The document retrieval module 204 is used to, after the overall model training is completed, receive the query text input by the user as the corresponding first query text; and use the currently user-specified document library as the corresponding first document library; and perform primary document selection on the first document library according to the first query text, the context capacity threshold, and the similarity scoring model to obtain a primary selected document sequence; and perform final document selection on the primary selected document sequence according to the first query text, the context capacity threshold, and the relevance scoring model to obtain a final selected document sequence and feedback it to the current user; the first document library includes a plurality of first documents; both the primary selected and final selected document sequences are sorted by a plurality of first documents.

[0192] A document retrieval device using a large language model as an encoder provided by an embodiment of the present invention can execute the method steps in the above method embodiment, and its implementation principle and technical effect are similar, and will not be elaborated here.

[0193] It should be noted that it should be understood that the division of each module of the above device is only a logical function division. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by a processing element; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the preprocessing module can be a separately established processing element, or can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a certain processing element of the above device to perform the functions of the above determined modules. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together or independently implemented. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit in the processor element or the instruction in the form of software.

[0194] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), etc. Again, when a certain module above is implemented in the form of a processing element scheduler code, the processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processors that can call program code. Again, these modules may be integrated together and implemented in the form of a System-on-a-chip (SOC).

[0195] In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the foregoing method embodiments are generated in whole or in part. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.). The above computer-readable storage medium may be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The above available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0196] Figure 4 FIG. is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server that implements the method of the foregoing embodiments, or a terminal device or a server that is connected to the foregoing terminal device or server and implements the method of the foregoing embodiments. As Figure 4As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver operations of the transceiver 303. Various instructions may be stored in the memory 302 for completing various processing functions and implementing the processing steps described in the method of the foregoing embodiments. Preferably, the electronic device according to the embodiment of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to implement communication connections between components. The above communication port 306 is used for the electronic device to connect and communicate with other peripherals.

[0197] As described in Figure 4 The system bus 305 mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to implement communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include a Random Access Memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0198] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), a Graphics Processing Unit (GPU), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0199] It should be noted that the embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored, and when they run on a computer, the computer is caused to execute the methods and processing procedures provided in the above embodiments.

[0200] An embodiment of the present invention provides a document retrieval method, apparatus, electronic device, and computer-readable storage medium using a large language model as an encoder. As can be seen from the above, the embodiment of the present invention selects a large language model that has been pre-trained and can process K-level contexts at one time as the query, document, and joint encoders; constructs a similarity scoring model with the query and document encoders as the core, and constructs a relevance scoring model with the joint encoder as the core; and trains the similarity and relevance scoring models respectively; when training, if the dataset is large, it is trained in the way of fine-tuning all model parameters, and if the dataset is small, to avoid overfitting, low-rank matrix parameters are implanted into the three types of encoders and trained by fine-tuning the low-rank matrix; after the overall training is completed, the similarity scoring model is used to retrieve documents from the document library specified by the user according to the query text input by the user to obtain a primary selected document sequence, and the relevance scoring model is used to finally select documents from the primary selected document sequence to obtain the corresponding final selected document sequence and feedback it to the current user. The embodiment of the present invention effectively improves the accuracy of the document retrieval task by introducing a pre-trained large language model.

[0201] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented in hardware, software modules executed by a processor, or a combination of both. The software modules may be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0202] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A document retrieval method using a large language model as an encoder, characterized in that, The method includes: Select a large language model that has been pre-trained and can process K-level contexts at once as three types of encoders: a query encoder, a document encoder, and a joint encoder; and record the maximum context capacity processed by the large language model at once as the corresponding context capacity threshold; the context capacity threshold is in thousands of characters; the large language model is preferably the LLaMA series model; Construct a similarity scoring model with the query and document encoders as the core; and construct a relevance scoring model with the joint encoder as the core; the similarity scoring model is used to perform similarity scoring processing on the query text and document text input to the model and output the corresponding similarity score; the relevance scoring model is used to perform relevance scoring processing on the query text and the document text input to the model and output the corresponding relevance score; Train the similarity and relevance scoring models respectively based on a preset first data set; the first data set includes multiple first data records; the first data record includes a training query text and a set of positive sample documents; the set of positive sample documents consists of multiple positive sample documents; the number of characters in each of the positive sample documents is less than or equal to the context capacity threshold; the two training query texts of any two of the first data records are not relevant; the training query text of each of the first data records is only relevant to each of the positive sample documents of the current first data record; After the overall model training is completed, receive the query text input by the user as the corresponding first query text; and use the currently user-specified document library as the corresponding first document library; and perform a preliminary document selection on the first document library according to the first query text, the context capacity threshold, and the similarity scoring model to obtain a preliminary document sequence; and perform a final document selection on the preliminary document sequence according to the first query text, the context capacity threshold, and the relevance scoring model to obtain a final document sequence and feedback it to the current user; the first document library includes multiple first documents; the preliminary and final document sequences are both sorted by multiple of the first documents.

2. The document retrieval method using a large language model as an encoder according to claim 1, wherein The first model input end of the similarity scoring model is used to receive the query text, the second model input end is used to receive the document text, and the model output end is used to output the similarity score; The similarity scoring model includes a first preprocessing module, the query encoder, the document encoder, and a similarity calculation module; the first and second input ends of the first preprocessing module are respectively connected to the first and second model input ends of the similarity scoring model, the first output end is connected to the input end of the query encoder, and the second output end is connected to the input end of the document encoder; the output ends of the query and document encoders are respectively connected to the first and second input ends of the similarity calculation module; the output end of the similarity calculation module is connected to the model output end of the similarity scoring model; The first preprocessing module is used to perform word segmentation processing on the query text and the document text input to the model according to the word segmentation rules of the large language model to obtain corresponding query word segmentation sequences and document word segmentation sequences; and perform embedding encoding processing on the query and document word segmentation sequences according to the word segmentation embedding encoding rules of the large language model to obtain corresponding query embedding encoding vectors and document embedding encoding vectors; and send the query embedding encoding vector to the query encoder and send the document embedding encoding vector to the document encoder; The query encoder is used to perform text semantic feature extraction processing on the input query embedding encoding vector to obtain a corresponding query feature vector and send it to the similarity calculation module; The document encoder is used to perform text semantic feature extraction processing on the input document embedding encoding vector to obtain a corresponding document feature vector and send it to the similarity calculation module; The similarity calculation module is used to calculate the similarity of the query and document feature vectors according to a preset vector similarity algorithm and output the calculation result as the corresponding similarity score; the vector similarity algorithm at least includes the cosine vector similarity algorithm; the score of the similarity score is between 0 and 1.

3. The document retrieval method using a large language model as an encoder according to claim 1, wherein The first model input end of the relevance scoring model is used to receive the query text, the second model input end is used to receive the document text, and the model output end is used to output the relevance score; The relevance scoring model includes a second preprocessing module, the joint encoder and the MLP model; the first and second input ends of the second preprocessing module are respectively connected to the first and second model input ends of the relevance scoring model, and the output end is connected to the input end of the joint encoder; the output end of the joint encoder is connected to the input end of the MLP model; the output end of the MLP model is connected to the model output end of the relevance scoring model; The second preprocessing module is used to perform word segmentation processing on the query text and the document text input to the model according to the word segmentation rules of the large language model to obtain corresponding query word segmentation sequences and document word segmentation sequences; and by adding a preset separator word 'SEP' between the query word segmentation sequence and the document word segmentation sequence, perform sequence connection on the query and document word segmentation sequences to obtain a corresponding joint word segmentation sequence = query word segmentation sequence ⊕ 'SEP' ⊕ document word segmentation sequence, where ⊕ is a string concatenation symbol; and perform embedding encoding processing on the joint word segmentation sequence according to the word segmentation embedding encoding rules of the large language model to obtain a corresponding joint embedding encoding vector and send it to the joint encoder; The joint encoder is used to perform text semantic feature extraction processing on the input joint embedding encoding vector to obtain a corresponding joint feature vector and send it to the MLP model; The MLP model is formed by connecting an input processing unit and an output processing unit; the input processing unit is formed by connecting one or more fully connected activation layers, and each of the fully connected activation layers is formed by connecting a fully connected layer and a non-linear activation function; the output processing unit is formed by connecting a fully connected layer, a softmax function, and an output layer; The MLP model is used to perform relevance prediction based on the joint feature vector and output the obtained prediction result as the corresponding relevance score. Specifically, the input processing unit takes the input vector feature space of the output processing unit as the corresponding target feature space and performs vector mapping of the joint feature vector in the target feature space to obtain a corresponding feature mapping vector and inputs it into the output processing unit; and the fully connected layer of the output processing unit performs a fully connected operation on the input feature mapping vector to obtain a fully connected vector with a feature dimension of 2, and the softmax function of the output processing unit performs binary classification probability calculation of relevant and irrelevant based on the fully connected vector to obtain corresponding relevant probability and irrelevant probability, and the output layer of the output processing unit outputs the relevant probability as the corresponding relevance score; the sum of the relevant probability and the irrelevant probability is 1; the score of the relevance score is between 0 and 1.

4. The document retrieval method using a large language model as an encoder according to claim 1, characterized in that, The similarity and relevance scoring models are respectively trained based on a preset first data set, specifically including: Step 41, count the total number of records in the first data set to obtain a corresponding first total; Step 42, identify whether the first total exceeds a preset total threshold; if the first total has exceeded the total threshold, set the corresponding first parameter adjustment mode to the first mode; if the first total does not exceed the total threshold, perform key module marking on the query, document, and joint encoder respectively according to the preset key module definition rules to obtain the corresponding query key module set, document key module set, and joint key module set, and implant a set of corresponding low-rank matrix parameters on the output weight parameters of each of the key modules in the query, document, and joint key module sets to obtain the corresponding low-rank matrix parameter sets G Q , G D and G Q+D , and set the corresponding first parameter adjustment mode to the second mode; Among them, each of the low-rank matrix parameters consists of a corresponding pair of low-rank matrices A and B; the original module output weight parameters of each of the key modules in the query, document, and joint key module sets are denoted as W0; the new module output weight parameters obtained after implanting the corresponding low-rank matrix parameters into each of the key modules in the query, document, and joint key module sets are denoted as W * = W0 + AB; the set of low-rank matrix parameters G Q corresponding to the query encoder consists of all the low-rank matrix parameters corresponding to the query key module set; the set of low-rank matrix parameters G D corresponding to the document encoder consists of all the low-rank matrix parameters corresponding to the document key module set; the set of low-rank matrix parameters G Q+D corresponding to the joint encoder consists of all the low-rank matrix parameters corresponding to the joint key module set; Step 43, train the similarity scoring model according to the first tuning mode and the first data set; Step 44, train the relevance scoring model according to the first tuning mode and the first data set; Step 45, after the model training of the similarity scoring model and the relevance scoring model is completed, confirm that the overall model training is completed.

5. The document retrieval method using a large language model as an encoder according to claim 4, wherein The training of the similarity scoring model according to the first tuning mode and the first data set specifically includes: Step 51, divide the first data set into two sub-data sets based on a preset first splitting ratio and denote them as corresponding first training set and first evaluation set; Among them, both the first training set and the first evaluation set are composed of multiple first data records; the ratio of the total number of records in the first training set to the first evaluation set satisfies the first splitting ratio; Step 52, record the total number of records in the first training set as the corresponding total N1; and count the total number of positive sample documents for each of the first data records in the first training set to obtain the corresponding first sample total, and calculate the sum of the N1 first sample totals as the corresponding total N2; and denote each of the training query texts in the first training set as query x i , where 1 ≤ index i ≤ N1; and denote each of the positive sample documents in the first training set as document d j , where 1 ≤ sample index j ≤ N2; and based on each of the queries x i and each of the documents d j set a corresponding label y i,j ; and from the N1 queries x i , the N2 documents d j and the N1×N2 labels y i,j form N1×N2 first data groups (x i , d j , y i,j ); Among them, the label y i,j takes a value of 0 or 1; The label y i,j is set as follows: If the query x i and the document d j belong to the same first data record, the corresponding label y i,j is set to 1; if the query x i and the document d j do not belong to the same first data record, the corresponding label y i,j is set to 0; Step 53, input the query x i , d j , y i,j ) of each of the first data groups (x i and the document d j as the corresponding query text and document text into the similarity scoring model for processing; and cache the query and the query and document feature vectors output by the document encoder during this processing as the corresponding vectors eq i,j and vector ed i,j ; and form a corresponding second data group (eq i,j , ed i,j , y i,j ) from the cached vector eq i,j , the vector ed i,j , and the corresponding label y i,j ; Step 54, bringing the obtained N1×N2 second data groups (eq i,j , ed i,j , y i,j ) into a preset model loss function L A ; and identifying the first parameter tuning mode; if the first parameter tuning mode is the first mode, then based on a preset first model optimizer, modulating the full model parameters of the query and document encoders of the similarity scoring model in one round in the direction of minimizing the model loss function L A ; if the first parameter tuning mode is the second mode, then based on a preset second model optimizer, modulating the low-rank matrix parameter sets G A and G Q corresponding to the query and document encoders in one round in the direction of minimizing the model loss function L D . Among them, the model loss function L A is implemented based on a contrastive loss function, specifically: f() is a preset distance metric function, used to calculate the vector distance between two vectors input to the function; the distance metric function at least includes cosine distance function, Euclidean distance function, Jaccard distance function, Hamming distance function; m is a preset distance threshold parameter; The first model optimizer at least includes Adam optimizer, SGD optimizer; The second model optimizer at least includes Adam optimizer, SGD optimizer; Step 55, record the total number of records in the first evaluation set as the corresponding total N3; and count the total number of positive sample documents for each of the first data records in the first evaluation set to obtain the corresponding second sample total, and calculate the sum of the N3 second sample totals as the corresponding total N4; and denote each of the training query texts in the first evaluation set as query x u , where 1 ≤ index u ≤ N3; and denote each of the positive sample documents in the first evaluation set as document d v , where 1 ≤ sample index v ≤ N4; and based on each query x u and each document d v set a corresponding label y u,v ; and from the N3 queries x u , the N4 documents d v and the N3×N4 labels y u,v form N3×N4 third data groups (x u , d v , y u,v ); Among them, the label y u,v takes a value of 0 or 1; The label y u,v is set as follows: If the query x u and the document d v belong to the same first data record, the corresponding label y u,v is set to 1; if the query x u and the document d v do not belong to the same first data record, the corresponding label y u,v is set to 0; Step 56, input the query x u , d v , y u,v ) of each of the third data groups (x u and the document d v as the corresponding query text and document text into the similarity scoring model for processing; and cache the query and the query and document feature vectors output by the document encoder during this processing as the corresponding vectors eq u,v and the vector ed u,v ; and form a corresponding fourth data group (eq u,v , ed u,v , y u,v ) from the cached vector eq u,v , the vector ed u,v , and the corresponding label y u,v ); Step 57, bringing the obtained N3×N4 fourth data groups (eq u,v , ed u,v , y u,v ) into the preset model evaluation function F A for calculation to obtain the corresponding first evaluation value; Among them, the model evaluation function F A is as follows: Step 58, identify whether the first evaluation value meets a preset first evaluation value range; if not, return to Step 53 to continue training; if so, confirm that the model training of the similarity scoring model is completed.

6. The document retrieval method using a large language model as an encoder according to claim 4, characterized in that The training of the relevance scoring model according to the first hyperparameter tuning mode and the first data set specifically includes: Step 61, divide the first data set into two sub-data sets denoted as the corresponding second training set and second evaluation set based on a preset second splitting ratio; Among them, both the second training set and the second evaluation set are composed of multiple first data records; the ratio of the total number of records in the second training set to the second evaluation set meets the second splitting ratio; Step 62, record the total number of records in the second training set as the corresponding total N5; and count the total number of the positive sample documents of each of the first data records in the second training set to obtain the corresponding third sample total, and calculate the sum of the N5 third sample totals as the corresponding total N6; and denote each of the training query texts in the second training set as query x i , where 1 ≤ index i ≤ N5; and denote each of the positive sample documents in the second training set as document d j , where 1 ≤ sample index j ≤ N6; and based on each of the queries x i and each of the documents d j set a corresponding label y i,j ; and from the N5 queries x i , the N6 documents d j and the N5×N6 labels y i,j form N5×N6 fifth data groups (x i , d j , y i,j ); Among them, the label y i,j takes a value of 0 or 1; The label y i,j is set as follows: If the query x i and the document d j belong to the same first data record, then the corresponding label y i,j is set to 1; if the query x i and the document d j do not belong to the same first data record, then the corresponding label y i,j is set to 0; Step 63, input the query x i , d j , y i,j ) of each of the fifth data groups (x i and the document d j as the corresponding query text and document text into the relevance scoring model for processing; and cache the relevant probability and irrelevant probability output by the softmax function of the MLP model inside the relevance scoring model during this processing as the corresponding probabilities pp i,j and probability pn i,j ; and form a corresponding sixth data group (pp i,j , pn i,j , y i,j ) from the cached probability pp i,j , the probability pn i,j , and the corresponding label y i,j ; Step 64, bringing the obtained N5×N6 sixth data groups (pp i,j , pn i,j , y i,j ) into a preset model loss function L B ; and identifying the first parameter tuning mode; if the first parameter tuning mode is the first mode, then based on a preset third model optimizer, modulating the overall model parameters of the joint encoder and the MLP model of the correlation scoring model in one round in the direction of minimizing the model loss function L B ; if the first parameter tuning mode is the second mode, then based on a preset fourth model optimizer, modulating the low-rank matrix parameter set G B corresponding to the joint encoder and the model parameters of the MLP model in one round in the direction of minimizing the model loss function L Q+D ; Among them, the model loss function L B is implemented based on the cross-entropy loss function, specifically as follows: The third model optimizer includes at least Adam optimizer and SGD optimizer; The fourth model optimizer includes at least Adam optimizer and SGD optimizer; Step 65, record the total number of records in the second evaluation set as the corresponding total number N7; and count the total number of the positive sample documents of each of the first data records in the second evaluation set to obtain the corresponding fourth sample total, and calculate the sum of the N7 fourth sample totals as the corresponding total number N8; and denote each of the training query texts in the second evaluation set as query x u , where 1 ≤ index u ≤ N7; and denote each of the positive sample documents in the second evaluation set as document d v , where 1 ≤ sample index v ≤ N8; and based on each query x u and each document d v , set a corresponding label y u,v ; and from the N7 queries x u , the N8 documents d v , and the N7×N8 labels y u,v , form N7×N8 seventh data groups (x u , d v , y u,v ); wherein, the label y u,v takes a value of 0 or 1; The label y u,v is set according to the following rule: if the query x u and the document d v belong to the same first data record, then the corresponding label y u,v is set to 1; if the query x u and the document d v do not belong to the same first data record, then the corresponding label y u,v is set to 0; Step 66, input the query x u , d v , y u,v ) of each of the seventh data groups (x u and the document d v as the corresponding query text and document text into the relevance scoring model for processing, and record the relevance score output by the model this time as the corresponding predicted score and form a corresponding eighth data group from the predicted score obtained this time i,j and the corresponding label y Step 67, bring the obtained N7×N8 eighth data groups into a preset model evaluation function F B for calculation to obtain corresponding second evaluation values; Among them, the model evaluation function F B is implemented based on the MAE function, the MSE function or the RMSE function; The model evaluation function F B When based on the MAE function, specifically: The model evaluation function F B When based on the MSE function, specifically: The model evaluation function F B When based on the RMSE function, specifically: Step 68, identify whether the second evaluation value meets a preset second evaluation value range; if not, return to Step 63 to continue training; if so, confirm that the model training of the relevance scoring model is completed.

7. The document retrieval method using a large language model as an encoder according to claim 1, characterized in that, The document pre-selection of the first document library according to the first query text, the context capacity threshold, and the similarity scoring model to obtain a pre-selected document sequence specifically includes: Take each first document in the first document library as the corresponding current document; and identify whether the number of words in the current document exceeds the context capacity threshold; if not, input the first query text and the current document as the corresponding query text and document text into the similarity scoring model for similarity scoring processing and use the similarity score obtained this time as the corresponding first document score; if so, perform sharding processing on the current document with the context capacity threshold as the maximum shard word count to obtain corresponding multiple first document fragments, and input the first query text and each first document fragment as the corresponding query text and document text into the similarity scoring model for similarity scoring processing and use the similarity score obtained this time as the corresponding first fragment score, and use the maximum score among all the first fragment scores corresponding to the current document as the corresponding first document score; Sort all the first documents in the first document library in descending order according to the first document score to obtain a corresponding first document sequence; and extract the first R first documents in the first document sequence to form the corresponding pre-selected document sequence; R is a preset positive integer value.

8. The document retrieval method using a large language model as an encoder according to claim 1, characterized in that The document final selection of the pre-selected document sequence according to the first query text, the context capacity threshold, and the relevance scoring model to obtain a final selected document sequence and feedback to the current user specifically includes: Take each of the first documents in the initial selection document sequence as the corresponding current document; identify whether the number of words in the current document exceeds the context capacity threshold; if not, input the first query text and the current document as the corresponding query text and document text into the relevance scoring model for relevance scoring processing, and take the obtained relevance score as the corresponding second document score for this time; if it has exceeded, perform fragmentation processing on the current document with the context capacity threshold as the maximum fragmentation word count to obtain corresponding multiple second document fragments, input the first query text and each of the second document fragments as the corresponding query text and document text into the relevance scoring model for relevance scoring processing, take the obtained relevance score as the corresponding second fragment score for this time, and take the maximum score among all the second fragment scores corresponding to the current document as the corresponding second document score; Sort all the first documents in the initial selection document sequence in descending order according to the second document score to obtain the corresponding second document sequence; remove the first documents in the second document sequence whose second document scores are lower than the preset relevance score threshold; and feedback the latest second document sequence to the current user as the corresponding final selection document sequence.

9. An apparatus for performing the document retrieval method using a large language model as an encoder according to any one of claims 1-8, characterized in that, The device includes: a preprocessing module, a model construction module, a model training module, and a document retrieval module; The preprocessing module is used to select a large language model that has completed pre-training and can process K-level contexts at one time as three types of encoders: a query encoder, a document encoder, and a joint encoder; and record the maximum context capacity processed by the large language model at one time as the corresponding context capacity threshold; the context capacity threshold is in units of one thousand words; the large language model preferably uses the LLaMA series of models; The model construction module is used to construct a similarity scoring model with the query and document encoders as the core; and construct a relevance scoring model with the joint encoder as the core; the similarity scoring model is used to perform similarity scoring processing on the query text and document text input into the model and output the corresponding similarity score; the relevance scoring model is used to perform relevance scoring processing on the query text and the document text input into the model and output the corresponding relevance score; The model training module is used to train the similarity and relevance scoring models respectively based on a preset first data set; the first data set includes multiple first data records; the first data record includes a training query text and a positive sample document set; the positive sample document set consists of multiple positive sample documents; the number of words in each of the positive sample documents is less than or equal to the context capacity threshold; the two training query texts of any two first data records are not relevant; the training query text of each first data record is only relevant to each positive sample document of the current first data record; The document retrieval module is configured to, after the overall model training is completed, receive the query text input by the user as the corresponding first query text; use the document library specified by the current user as the corresponding first document library; perform a primary document selection on the first document library according to the first query text, the context capacity threshold, and the similarity scoring model to obtain a primary selected document sequence; and perform a final document selection on the primary selected document sequence according to the first query text, the context capacity threshold, and the relevance scoring model to obtain a final selected document sequence and feedback it to the current user; the first document library includes a plurality of first documents; both the primary selected and final selected document sequences are sorted by a plurality of the first documents.

10. An electronic device, characterized in that, Comprising: A memory, a processor, and a transceiver; The processor is configured to be coupled with the memory, read and execute the instructions in the memory to implement the method according to any one of claims 1-8; The transceiver is coupled with the processor, and the processor controls the transceiver to perform message sending and receiving.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1-8.

Images