Retrieval sorting method and device based on large language model

Abstract

The embodiment of the invention relates to a retrieval sorting method and device based on a large language model. The method comprises the following steps: selecting a large language model which is realized based on a coder-decoder structure and is pre-trained as a first LLM (Logical Language Model); constructing a document scoring / sorting model based on the first LLM model; training a document scoring / sorting model on the basis of a preset score / data set under the condition that model parameters of the first LLM model are unchanged; after training is finished, receiving a first query text input by the user, and taking a document library specified by the user as a first document library; performing first-stage document screening on the first document library according to the first query text in a word frequency scoring mode; performing two-stage document screening on the primarily screened document sequence by using a document scoring model according to the first query text; and performing three-stage document reordering on the secondary screening document sequence by using a document ordering model according to the first query text to obtain a recommended document sequence and feeding back the recommended document sequence to the user. According to the invention, the retrieval precision can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly relates to a retrieval and sorting method and device based on a large language model. Background Art

[0002] Traditional text retrieval systems usually retrieve and sort information based on the word frequency density of query keywords. However, in practical applications, we found that the conventional retrieval and sorting methods are not ideal when dealing with document-level retrieval tasks. This is mainly because the document content has a large number of words and complex context semantics. If only relying on conventional methods for retrieval and sorting, on the one hand, the range of retrieval results may be too broad, and on the other hand, the sorting results are also difficult to accurately reflect the relevance between the document and the query. Generally speaking, when dealing with document-level retrieval tasks, the conventional retrieval and sorting methods are prone to problems of insufficient retrieval accuracy.

[0003] Large Language Models (LLMs) have relatively strong context storage capabilities and context semantic understanding capabilities. If a pre-trained large language model can be introduced into the retrieval and sorting tasks, it will surely effectively improve the retrieval accuracy. And how to introduce a large language model to handle the retrieval and sorting tasks is the technical problem to be solved by the present invention. Summary of the Invention

[0004] The object of the present invention is to provide a retrieval and sorting method, device, electronic device and computer-readable storage medium based on a large language model in view of the defects of the prior art. The present invention selects a large language model based on the encoder-decoder structure of the Transformer framework and has completed pre-training as the first LLM model; and constructs two task models (document scoring model, document sorting model) by configuring a downstream MLP model on the output side of the first LLM model; and trains the two task models separately under the condition that the model parameters of the first LLM model remain unchanged; and uses the two task models to improve the retrieval accuracy after the training is completed, specifically: first, perform a first-stage document screening on the user-specified document library according to the query text input by the user in the word frequency scoring manner to obtain a preliminary screening document sequence, then use the document scoring model to perform a second-stage document screening on the preliminary screening document sequence according to the query text to obtain a secondary screening document sequence, and finally use the document sorting model to perform a third-stage document re-sorting on the secondary screening document sequence according to the query text to obtain a corresponding recommended document sequence and feedback it to the current user. When processing the retrieval and sorting task, the present invention first performs full-text relevant document retrieval and sorting on the target library through a conventional sorting method, then performs secondary document screening and sorting on the preliminary screening document sequence generated by the conventional sorting method through the document scoring model, and finally performs a re-sorting on the secondary screening result, that is, the secondary screening document sequence, according to the relevance level again through the document sorting model. Through the present invention, the range of retrieved documents can be effectively narrowed, and the relevance between the document and the query can be captured more accurately, so as to achieve the purpose of improving the retrieval accuracy.

[0005] To achieve the above object, in the first aspect of the embodiments of the present invention, a retrieval and sorting method based on a large language model is provided, and the method includes:

[0006] Select a large language model based on the encoder-decoder structure of the Transformer framework and having completed pre-training as the first LLM model; and construct two task models denoted as a document scoring model and a document sorting model by configuring a downstream MLP model on the output side of the first LLM model; the document scoring model is used to predict the relevance score between the input query Q and the document D of the model and output the corresponding score S; the document sorting model is used to sort the input document sequence {D j} according to the input query Q of the model and output the corresponding sorting sequence {Id c}; the document sequence {D j} is composed of multiple documents D j sorted, 1 ≤ document index j ≤ N D where N D is the total number of documents in the document sequence {D j}; the sorting sequence {Id c} is composed of ND a document number Id c composed, where 1 ≤ sorting rank c ≤ N D , and each of the said document numbers Id c corresponds to a said document index j; in the said sorting sequence {Id c}, the higher the sorting rank, the higher the matching degree of the corresponding document to the query;

[0007] Under the condition that the model parameters of the first LLM model remain unchanged, the document scoring model and the document sorting model are respectively trained based on a preset scoring data set and sorting data set;

[0008] After the training of both models is completed, the query text input by the user is received as the corresponding first query text; and the document library specified by the current user is used as the corresponding first document library; the first document library includes multiple first documents;

[0009] According to the word frequency scoring method, a first-stage document screening is performed on the first document library according to the first query text to obtain a corresponding preliminary screening document sequence;

[0010] The document scoring model is used to perform a second-stage document screening on the preliminary screening document sequence according to the first query text to obtain a corresponding secondary screening document sequence;

[0011] The document sorting model is used to perform a third-stage document re-sorting on the secondary screening document sequence according to the first query text to obtain a corresponding recommended document sequence and feedback it to the current user.

[0012] Preferably, the first LLM model is composed of a first encoder and a first decoder connected; the first encoder is implemented based on the Encoder model structure of the Transformer framework, and the first decoder is implemented based on the Decoder model structure of the Transformer framework;

[0013] The first model input end of the document scoring model is used to receive the query Q, the second model input end is used to receive the document D, and the model output end is used to output the corresponding score S; the document scoring model includes a first preprocessing module, the first LLM model and a first MLP model; the first and second input ends of the first preprocessing module are connected to the first and second model input ends of the document scoring model, and the output end is connected to the input end of the first LLM model; the output end of the first LLM model is connected to the input end of the first MLP model; the output end of the first MLP model is connected to the model output end of the document scoring model;

[0014] The first model input end of the document sorting model is used to receive the query Q, and the second model input end is used to receive the document sequence {D j}, and the model output end is used to output the corresponding sorting sequence {Id j}; the document sorting model includes a second preprocessing module, the first LLM model, a segmentation mapping module, a second MLP model, and a sorting output module; the first and second input ends of the second preprocessing module are connected to the first and second model input ends of the document sorting model, the first output end is connected to the input end of the first LLM model, and the second output end is connected to the first input end of the segmentation mapping module; the output end of the first LLM model is connected to the second input end of the segmentation mapping module; the output end of the segmentation mapping module is connected to the input end of the second MLP model; the output end of the second MLP model is connected to the input end of the sorting output module; the output end of the sorting output module is connected to the model output end of the document sorting model;

[0015] The scoring data set includes a first training query set, a first training document set, and a first label set. The first training query set includes multiple first training queries, the first training document set includes multiple first training documents, and the first label set includes multiple first relevance labels; the value of the first relevance label is 0 or 1; each first relevance label is used to mark the relevance of any group (first training query, first training document), and a value of 0 indicates irrelevance, and a value of 1 indicates relevance;

[0016] The sorting data set includes multiple first data records; the first data record includes a second training query, a second training document sequence, and a first label sequence; the second training document sequence is sorted by multiple second training documents, and each second training document corresponds to a sequence index value; the first label sequence is composed of multiple first label numbers, the total number of the first label numbers matches the total number of the second training documents, and the number value of each first label number corresponds to the sequence index value of a second training document.

[0017] Further, the first preprocessing module is used to perform word segmentation processing on the query Q and the document D input to the model according to the word segmentation rules corresponding to the first LLM model to obtain corresponding first word segmentation sequences and second word segmentation sequences; and by adding a preset separator word 'SEP' between the first and second word segmentation sequences, perform sequence connection on the first and second word segmentation sequences to obtain a corresponding third word segmentation sequence = first word segmentation sequence ⊕ 'SEP' ⊕ second word segmentation sequence, where ⊕ is a string concatenation symbol; and perform embedding encoding processing on the third word segmentation sequence according to the embedding encoding rules of the first LLM model to obtain a corresponding embedding vector X{xi}; and use the vector length of the embedding vector X{x i} as the initial vector length, and use the output vector feature dimension of the first decoder of the first LLM model as the initial vector feature dimension, and set a all-zero vector as the corresponding initial decoding vector Y0{y 0,i}; and send the embedding vector X{x i} and the initial decoding vector Y0{y 0,i} to the first LLM model; the embedding vector X{x i} is composed of multiple embedding sub-vectors x i sorted in order, 1 ≤ sub-vector index i ≤ N1, N1 is the total number of word segments of the third word segment sequence, and the embedding sub-vector x i corresponds to each word segment of the third word segment sequence one by one; the initial decoding vector Y0{y 0,i} includes N1 initial decoding sub-vectors y 0,i , and the initial decoding sub-vector y 0,i corresponds to the embedding sub-vector x i one by one;

[0018] The first encoder of the first LLM model is used to perform high-dimensional feature encoding on the embedding vector X{x i} to obtain the corresponding encoded vector E{e i} and send it to the first decoder; the first decoder of the first LLM model is used to use the initial decoding vector Y0{y 0,i} as the initial state quantity of the decoding vector and perform multiple loop iterations of decoding according to the encoded vector E{e i} to obtain the corresponding decoding vector Y{y i} and send it to the first MLP model; the vector length and vector feature dimension of the decoding vector Y{y i} are consistent with the initial decoding vector Y0{y 0,i}, and the decoding vector Y{y i} includes N1 decoding sub-vectors y i , and the decoding sub-vector y i corresponds to the initial decoding sub-vector y 0,i one by one;

[0019] The first MLP model is formed by connecting a first input layer, a first hidden layer, and a first output layer; the first input layer consists of a single fully-connected layer; the first hidden layer consists of one or more first fully-connected activation layers, and each of the first fully-connected activation layers is formed by connecting a fully-connected layer and a ReLU activation function layer; the first output layer is formed by connecting a fully-connected layer and a Sigmoid activation function layer; the output data of the first output layer is a probability data output by the Sigmoid activation function layer;

[0020] The first MLP model is used to input the decoded vector Y{y i} into the first input layer, and through the layer-by-layer operations of the first input layer, the first hidden layer, and the first output layer, obtain the probability data output by the Sigmoid activation function layer, and take the probability data obtained in that instance as the corresponding score S to output; the value range of the score S is between 0 and 1.

[0021] Further, the second preprocessing module is used to count the total number of documents in the document sequence {D j} and take the statistical result as the corresponding total number of documents N D ; and perform word segmentation on the query Q input to the model according to the word segmentation rule corresponding to the first LLM model to obtain the corresponding fourth word segmentation sequence; and perform word segmentation on each document D j in the document sequence {D j} to obtain the corresponding fifth word segmentation sequence; and perform sequence connection on the fourth word segmentation sequence and each of the fifth word segmentation sequences through the separator word segmentation 'SEP' to obtain the corresponding sixth word segmentation sequence = fourth word segmentation sequence ⊕ 'SEP' ⊕ fifth word segmentation sequence; and perform sequential connection on N D sixth word segmentation sequences according to the sorting order of the document D j to obtain the corresponding seventh word segmentation sequence = sixth word segmentation sequence ⊕ '\n' ⊕ sixth word segmentation sequence ⊕... ⊕ '\n' ⊕ sixth word segmentation sequence; and perform embedding encoding processing on the seventh word segmentation sequence according to the embedding encoding rule of the first LLM model to obtain the corresponding embedding vector X{x k}; and use the vector length of the embedding vector X{x k} as the initial vector length and the output vector feature dimension of the first decoder of the first LLM model as the initial vector feature dimension, and set a all-zero vector as the corresponding initial decoded vector Y0{y 0,k}; and input the embedding vector X{x k} Denote the sub - vector index \(k\) corresponding to each newline character token '\n' in it as the corresponding segmentation marker position, and form the corresponding position set \(G\) from all the obtained segmentation marker positions; and send the embedding vector \(X\{x\) k} and the initial decoding vector \(Y_0\{y\) 0,k} to the first LLM model; and send the total number of documents \(N\) D and the position set \(G\) to the segmentation mapping module; The embedding vector \(X\{x\) k} is composed of multiple embedding sub - vectors \(x\) k sorted in order, \(1\leq\) sub - vector index \(k\leq N_2\), \(N_2\) is the total number of tokens in the seventh token sequence, and the embedding sub - vector \(x\) k corresponds one - to - one with the tokens in the seventh token sequence; The initial decoding vector \(Y_0\{y\) 0,k} includes \(N_2\) initial decoding sub - vectors \(y\) 0,k and the initial decoding sub - vector \(y\) 0,k corresponds one - to - one with the embedding sub - vector \(x\) k ;

[0022] The first encoder of the first LLM model is used to perform high - dimensional feature encoding on the embedding vector \(X\{x\) k} to obtain the corresponding encoded vector \(E\{e\) k} and send it to the first decoder; The first decoder of the first LLM model is used to use the initial decoding vector \(Y_0\{y\) 0,k} as the initial state quantity of the decoding vector and perform multiple loop iterations of decoding according to the encoded vector \(E\{e\) k} to obtain the corresponding decoding vector \(Y\{y\) k} and send it to the segmentation mapping module; The vector length and vector feature dimension of the decoding vector \(Y\{y\) k} are consistent with those of the initial decoding vector \(Y_0\{y\) 0,k}, and the decoding vector \(Y\{y\) k} includes \(N_2\) decoding sub - vectors \(y\) k and the decoding sub - vector \(y\) k corresponds one - to - one with the initial decoding sub - vector \(y\) 0,k ;

[0023] The segmentation mapping module is used to sequentially split the decoding vector \(Y\{y\) D} into \(N\) k} fragment decoding vectors according to all the segmentation marker positions in the total number of documents \(N\) D and the position set \(G\) and pad each variable - length fragment decoding vector with zeros Complete a fixed-length segment decoding vector with a vector length equal to the preset vector length L And use the input vector feature space of the second input layer of the second MLP model as the target feature space, and project each of the fixed-length segment decoding vectors based on a built-in MLP model to obtain a fixed-length segment projection vector h by performing vector projection in the target feature space j ; and by N D such segment projection vectors h j form the corresponding projection vector H{h j} and send it to the second MLP model; the segment decoding vector corresponds one-to-one with the sixth token sequence in the seventh token sequence;

[0024] The second MLP model is composed of the second input layer, the second hidden layer, and the second output layer connected in sequence; the second input layer consists of a single fully-connected layer; the second hidden layer consists of one or more second fully-connected activation layers, and each second fully-connected activation layer is composed of a fully-connected layer and a ReLU activation function connected in sequence; the second output layer consists of a fully-connected layer and a Softmax function layer; the fully-connected layer of the second output layer outputs a process vector with a vector length of N D and a vector feature dimension also of N D ; the Softmax function layer of the second output layer is used to regard the current process vector as an N D ×N D two-dimensional matrix and calculate the probability distribution of N D ranking positions for each row of this two-dimensional matrix to obtain a probability sub-vector p composed of N D ranking probability data j ; and by the obtained N D such probability sub-vectors p j form a corresponding probability vector P{p j};

[0025] The second MLP model is used to input the projection vector H{h j} into the second input layer, and through the sequential operations of the second input layer, the second hidden layer, and the second output layer, obtain the probability vector P{p j} output by the Softmax function layer and send it to the sorting output module; the probability vector P{p j} includes N D such probability sub-vectors p j ; the probability sub-vector p j corresponds one-to-one with the document D j ; the probability sub-vector pj Composed of N D pieces of the ranking probability data, which are in one-to-one correspondence with the sorting ranking c;

[0026] The sorting output module is used to convert the probability vector P{p j}} into a two-dimensional matrix A of N D ×N D , where each row of the two-dimensional matrix A corresponds to one of the probability sub-vectors p j , each column corresponds to one of the sorting rankings c, and each matrix element matches the ranking probability data corresponding to the current row and the current column; and the document index j corresponding to the matrix element with the largest probability value in each column of the two-dimensional matrix A is used as the document number Id corresponding to the current column c ; and the N D obtained in the order of the sorting ranking c from 1 to N D pieces of the document numbers Id c are sorted to obtain the corresponding sorting sequence {Id c} and output.

[0027] Preferably, separately training the document scoring model and the document sorting model based on a preset scoring data set and a sorting data set specifically includes:

[0028] Training the first MLP model of the document scoring model based on the scoring data set; and training the piecewise mapping module and the second MLP model of the document sorting model based on the sorting data set; and confirming that the training of both models is completed after the training of the document scoring model and the document sorting model is completed.

[0029] Further, training the first MLP model of the document scoring model based on the scoring data set specifically includes:

[0030] Step 61, counting the total number of the first training queries in the scoring data set to obtain the corresponding total number M1, and counting the total number of the first training documents to obtain the corresponding total number M2; and denoting each of the first training queries as the corresponding query q u , denoting each of the first training documents as the corresponding document d v , denoting each of the first relevance labels as the corresponding label score and forming M1×M2 first data groups from M1 queries q u , M2 documents d v and M1×M2 label scores ;

[0031] Step 62, for each of the first data groups the query q u and the document d v are used as the corresponding query Q and document D and input into the similarity scoring model for correlation scoring prediction, and the score S obtained from this prediction is used as the corresponding prediction score and a corresponding prediction score from each of the first data groups and the label score are combined to form a corresponding first prediction-label pair

[0032] Step 63, the M1×M2 first prediction-label pairs obtained are input into a preset first model loss function L A ; and based on a preset first model optimizer, the model parameters of the first MLP model of the document scoring model are modulated once in the direction of minimizing the first model loss function L A ;

[0033] wherein, the first model loss function L A is implemented based on a binary cross-entropy loss function, specifically:

[0034]

[0035] the first model optimizer includes at least an Adam optimizer and an SGD optimizer;

[0036] Step 64, at the end of this round of modulation, for each of the first data groups (q u , d v , a u,v ), the query q u and the document d v are used as the corresponding query Q and document D and input into the similarity scoring model for correlation scoring prediction, and the score S obtained from this prediction is used as the corresponding prediction score and a corresponding prediction score from each of the first data groups and the label score are combined to form a corresponding second prediction-label pair and the M1×M2 second prediction-label pairs obtained are input into a preset first model evaluation function F A for calculation to obtain a corresponding first evaluation value;

[0037] wherein, the first model evaluation function FA Implemented based on the RMSE function, specifically as follows:

[0038]

[0039] Step 65, identify whether the first evaluation value meets a preset first evaluation value range; if not, return to Step 62 to continue training; if so, confirm that the model training of the document scoring model ends.

[0040] Further, the training of the segment mapping module and the second MLP model of the document ranking model based on the ranking data set specifically includes:

[0041] Step 71, split the ranking data set into two sub-data sets denoted as the corresponding first training set and first evaluation set based on a preset first splitting ratio;

[0042] Among them, both the first training set and the first evaluation set are composed of multiple first data records; the ratio of the total number of records in the first training set and the first evaluation set meets the first splitting ratio;

[0043] Step 72, denote the total number of records in the first training set as M3; and regard each first data record in the first training set as the corresponding current training record; and denote the second training query of the current training record as the corresponding query q r , 1 ≤ index r ≤ M3; and count the total number of documents in the second training document sequence of the current training record to obtain the corresponding total number M r,4 , and denote the current second training document sequence as the corresponding document sequence {d r,w}, 1 ≤ index w ≤ M r,4 ; and denote the first label sequence of the current training record as the corresponding current label ranking sequence; and set a label vector with both the vector length and vector feature dimension being M r,4

[0044] Among them, the document sequence {d r,w} is composed of the corresponding M r,4 documents d r,w ;

[0045] The label vector is composed of the corresponding M r,4 label sub-vectors ; The label sub-vector corresponds one-to-one with the document d r,w ; The label sub-vector is composed of the corresponding M​r,4 One label probability data Composed, where 1 ≤ index g ≤ M r,4 , and each of the label probability data Corresponds to a ranking position in the current label sorting sequence; each of the label sub-vectors Only one of the label probability data Is 1, and the remaining M r,4 -1 label probability data Are all 0, and the label probability data that is 1 Corresponds to the document d corresponding to the current label sub-vector r,w Corresponds to the ranking position in the current label sorting sequence;

[0046] Step 73, use each query q in the first training set r And its corresponding document sequence {d r,w} as the corresponding query Q and document sequence {D j} and input them into the document ranking model for ranking processing; and during this processing, the probability vector P{p j} output by the second MLP model of the document ranking model is used as the corresponding prediction vector And form a corresponding third prediction-label pair from each label vector in the first training set And its corresponding prediction vector

[0047] Among them, the prediction vector Is composed of corresponding M r,4 Prediction sub-vectors ; the prediction sub-vector Corresponds to the document d r,w One by one; the prediction sub-vector Is composed of corresponding M r,4 Prediction probability data ;

[0048] Step 74, substitute the obtained M3 third prediction-label pairs Into the preset second model loss function L B ; and based on the preset second model optimizer, modulate the model parameters of the piecewise mapping module and the second MLP model of the document ranking model in the direction of minimizing the second model loss function L B Once; A

[0049] Among them, the first model loss function L AImplemented based on the multi-class cross-entropy loss function, specifically as follows:

[0050]

[0051] The second model optimizer includes at least an Adam optimizer and an SGD optimizer;

[0052] Step 75, record the total number of records in the first evaluation set as M5; and take each of the first data records in the first evaluation set as the corresponding current evaluation record; and denote the second training query of the current evaluation record as the corresponding query q l , 1 ≤ index l ≤ M5; and count the total number of documents in the second training document sequence of the current evaluation record to obtain the corresponding total M l,6 , and denote the current second training document sequence as the corresponding document sequence {d l,s}, 1 ≤ index s ≤ M l,6 ; and denote the first label sequence of the current evaluation record as the corresponding current label sorting sequence; and set a label vector with both the vector length and the vector feature dimension being M l,6

[0053] Among them, the document sequence {d l,s} consists of the corresponding M l,6 documents d l,s ;

[0054] The label vector consists of the corresponding M l,6 label sub-vectors ; the label sub-vector corresponds one-to-one with the document d l,s ; the label sub-vector consists of the corresponding M l,6 label probability data , 1 ≤ index t ≤ M l,6 , each label probability data corresponds to a ranking in the current label sorting sequence; among the label sub-vectors only one label probability data is 1, and the remaining M l,6 -1 label probability data are all 0, and the label probability data that is 1 corresponds to the ranking of the document d l,s corresponding to the current label sub-vector in the current label sorting sequence;

[0055] ​Step 76. For each query q in the first evaluation set l and its corresponding document sequence {d l,s}, use them as the corresponding query Q and document sequence {D j} and input them into the document ranking model for ranking processing; and during this processing, use the probability vector P{p j} output by the second MLP model of the document ranking model as the corresponding prediction vector And form a corresponding fourth prediction-label pair from each label vector in the first evaluation set and its corresponding prediction vector

[0056] Among them, the prediction vector is composed of corresponding M l,6 prediction sub-vectors ; the prediction sub-vector corresponds one-to-one with the document d l,s ; the prediction sub-vector is composed of corresponding M l,6 prediction probability data ;

[0057] Step 77. Bring the obtained M5 fourth prediction-label pairs into the preset second model evaluation function F B for calculation to obtain the corresponding second evaluation value;

[0058] Among them, the second model evaluation function F B is implemented based on the RMSE function, specifically:

[0059]

[0060] Step 78. Identify whether the second evaluation value meets the preset second evaluation value range; if not, return to Step 72 to continue training; if so, confirm that the model training of the document ranking model is completed.

[0061] Preferably, the word frequency scoring method performs a first-stage document screening on the first document library according to the first query text to obtain a corresponding preliminary screening document sequence, specifically including:

[0062] Step 81. Perform keyword extraction processing on the first query text to obtain a corresponding keyword set; and record each first document in the first document library as the corresponding document D z ;

[0063] Among them, the keyword set consists of one or more keywords kw​m Composed, where 1 ≤ index m ≤ N KEY , N KEY is the total number of keywords in the keyword set; 1 ≤ index z ≤ N Z , N Z is the total number of documents in the first document library;

[0064] Step 82, based on the keyword set and each document D z calculate the corresponding first-stage relevance score sm z ;

[0065] Among them, the calculation method of the first-stage relevance score sm z is as follows:

[0066]

[0067]

[0068] TF(kw m , D z ) is the term frequency of the current keyword kw m in the current document D z ; IDF(kw m ) is the inverse document frequency of the current keyword kw m in the first document library; k1 and b are two preset adjustment parameters; |D z | is the document length of the current document D z ; avgdl is the average document length of all the first documents in the first document library; n(kw m ) is the total number of the first documents containing the current keyword kw m in the first document library;

[0069] Step 83, sort the first documents in the first document library with the first-stage relevance score sm z higher than a preset first scoring threshold in descending order of the first-stage relevance score sm z to obtain the corresponding initial screening document sequence.

[0070] Preferably, using the document scoring model to perform second-stage document screening on the initial screening document sequence according to the first query text to obtain the corresponding secondary screening document sequence, specifically including:

[0071] Input the first query text and each of the first documents in the initially screened document sequence as the corresponding query Q and document D into the document scoring model for relevance scoring prediction, and use the score S output by the prediction for the current time as the corresponding two-stage relevance score; and sort the first documents in the initially screened document sequence with two-stage relevance scores higher than a preset second scoring threshold in descending order of the two-stage relevance scores to obtain the corresponding secondarily screened document sequence.

[0072] Preferably, using the document ranking model to perform three-stage document re-ranking on the secondarily screened document sequence according to the first query text to obtain the corresponding recommended document sequence and feedback it to the current user, specifically including:

[0073] Input the first query text and the secondarily screened document sequence as the corresponding query Q and document sequence {D j} into the document ranking model for ranking, and use the ranking sequence {Id c} output by the ranking for the current time as the corresponding current ranking sequence; and re-rank all the first documents in the secondarily screened document sequence according to the document number sorting order of the current ranking sequence to obtain the corresponding recommended document sequence and feedback it to the current user.

[0074] In the second aspect of the embodiments of the present invention, there is provided an apparatus for implementing the retrieval and ranking method based on a large language model described in the first aspect above. The apparatus includes: a model construction module, a model training module, a retrieval information receiving module, a first-stage retrieval and ranking module, a second-stage retrieval and ranking module, and a third-stage retrieval and ranking module;

[0075] The model construction module is used to select a large language model based on the encoder-decoder structure of the Transformer framework and having completed pre-training as the first LLM model; and construct two task models, denoted as the document scoring model and the document ranking model, by configuring a downstream MLP model on the output side of the first LLM model; the document scoring model is used to predict the relevance score of the query Q and the document D input into the model and output the corresponding score S; the document ranking model is used to rank the document sequence {D j} input into the model according to the query Q input into the model and output the corresponding ranking sequence {Id c}; the document sequence {D j} is composed of multiple documents D j sorted, 1 ≤ document index j ≤ N D , N D being the total number of documents in the document sequence {D j}; the ranking sequence {Id c} is composed of ND A document number Id c Consisting of, 1 ≤ sorting rank c ≤ N D , each of the said document number Ids c corresponds to a said document index j; The higher the sorting rank in the said sorting sequence {Id c}}, the higher the matching degree of the corresponding document to the query;

[0076] The said model training module is used to train the said document scoring model and the said document sorting model respectively based on a preset scoring data set and sorting data set under the condition that the model parameters of the said first LLM model remain unchanged;

[0077] The said retrieval information receiving module is used to receive the query text input by the user as the corresponding first query text after the training of both models is completed; and use the document library specified by the current user as the corresponding first document library; The said first document library includes a plurality of first documents;

[0078] The said first-stage retrieval and sorting module is used to perform a first-stage document screening on the said first document library according to the said first query text in a word frequency scoring manner to obtain a corresponding preliminary screening document sequence;

[0079] The said second-stage retrieval and sorting module is used to perform a second-stage document screening on the said preliminary screening document sequence according to the said first query text using the said document scoring model to obtain a corresponding secondary screening document sequence;

[0080] The said third-stage retrieval and sorting module is used to perform a third-stage document re-sorting on the said secondary screening document sequence according to the said first query text using the said document sorting model to obtain a corresponding recommended document sequence and feedback it to the current user.

[0081] The third aspect of the embodiments of the present invention provides an electronic device, including: a memory, a processor and a transceiver;

[0082] The said processor is used to be coupled with the said memory, read and execute the instructions in the said memory to implement the method steps described in the first aspect above;

[0083] The said transceiver is coupled with the said processor, and the said processor controls the said transceiver to perform message sending and receiving.

[0084] The fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, the computer-readable storage medium stores computer instructions, when the computer instructions are executed by a computer, the computer is caused to execute the instructions of the method described in the first aspect above.

[0085] An embodiment of the present invention provides a retrieval and ranking method, device, electronic device, and computer-readable storage medium based on a large language model. From the above content, it can be seen that the embodiment of the present invention selects a large language model based on the encoder-decoder structure of the Transformer framework and has completed pre-training as the first LLM model; and constructs two task models (document scoring model, document ranking model) by configuring a downstream MLP model on the output side of the first LLM model; and trains the two task models separately under the condition that the model parameters of the first LLM model remain unchanged; and uses the two task models to improve the retrieval accuracy after the training is completed. Specifically: first, perform a first-stage document screening on the user-specified document library according to the query text input by the user in the word frequency scoring method to obtain a preliminary screening document sequence, then use the document scoring model to perform a second-stage document screening on the preliminary screening document sequence again according to the query text to obtain a secondary screening document sequence, and finally use the document ranking model to perform a third-stage document re-ranking on the secondary screening document sequence according to the query text to obtain a corresponding recommended document sequence and feedback it to the current user. When processing the retrieval and ranking task, the embodiment of the present invention first performs full-library relevant document retrieval and ranking on the target library through a conventional ranking method, then performs secondary document screening and ranking on the preliminary screening document sequence generated by the conventional ranking method through the document scoring model, and finally performs another re-ranking on the secondary screening result, that is, the secondary screening document sequence, according to the relevance level through the document ranking model. The embodiment of the present invention achieves the effect of converging the retrieval document range, improves the relevance recognition accuracy between the document and the query, and improves the retrieval accuracy of the document-level retrieval task. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 FIG. is a schematic diagram of a retrieval and ranking method based on a large language model provided by Embodiment 1 of the present invention;

[0087] Figure 2 FIG. is a module schematic diagram of the document scoring model and the document ranking model provided by Embodiment 1 of the present invention;

[0088] Figure 3 FIG. is a module structure diagram of a retrieval and ranking device based on a large language model provided by Embodiment 2 of the present invention;

[0089] Figure 4 FIG. is a schematic diagram of the structure of an electronic device provided by Embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0090] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0091] Embodiment 1 of the present invention provides a retrieval and ranking method based on a large language model, as Figure 1 shown in the schematic diagram of a retrieval and ranking method based on a large language model provided by Embodiment 1 of the present invention. The method mainly includes the following steps:

[0092] Step 1, select a large language model implemented with an encoder-decoder structure based on the Transformer framework and having completed pre-training as the first LLM model; and construct two task models, denoted as the document scoring model and the document ranking model, by configuring a downstream MLP model on the output side of the first LLM model.

[0093] Here, the first LLM model of the embodiment of the present invention can be any large language model implemented with an encoder-decoder structure based on the Transformer framework and having completed pre-training, such as the T5 series models, BART series models, etc. Simply put: the first LLM model is composed of a first encoder and a first decoder connected; the first encoder is implemented based on the Encoder model structure of the Transformer framework, and the first decoder is implemented based on the Decoder model structure of the Transformer framework.

[0094] The document scoring model of the embodiment of the present invention is used to predict the relevance score between the query Q and the document D input to the model and output the corresponding score S.

[0095] The document ranking model of the embodiment of the present invention is used to rank the document sequence {D j} input to the model according to the query Q input to the model and output the corresponding ranking sequence {Id c}. Among them, the document sequence {D j} is sorted by multiple documents D j , 1 ≤ document index j ≤ N D , N D is the total number of documents in the document sequence {D j}; the ranking sequence {Id c} is composed of N D document numbers Id c , 1 ≤ ranking position c ≤ N D , each document number Id c corresponds to a document index j; the ranking sequence {Idc}, the higher the matching degree between the corresponding document and the query is as the ranking in it is higher. For example, when N D = 3, for the document sequence {D j}, the three documents D j＝1、2、3 are document A, B, and C respectively. If the relevance between query Q and document A is the worst and the relevance between query Q and document C is the highest, then, for the sorting sequence {Id c}, the three document numbers Id c＝1、2、3 are 3, 2, and 1 in sequence.

[0096] As Figure 2 shown in the module schematic diagram of the document scoring model and the document sorting model provided in the first embodiment of the present invention, the first model input end of the document scoring model is used to receive query Q, the second model input end is used to receive document D, and the model output end is used to output the corresponding score S. The internal model components of this document scoring model include: a first preprocessing module, a first LLM model, and a first MLP model.

[0097] The connection relationship of the internal components of the document scoring model is: the first and second input ends of the first preprocessing module are connected to the first and second model input ends of the document scoring model, and the output end is connected to the input end of the first LLM model; the output end of the first LLM model is connected to the input end of the first MLP model; the output end of the first MLP model is connected to the model output end of the document scoring model.

[0098] The functions of the internal components of the document scoring model are as follows:

[0099] 1) The first preprocessing module:

[0100] The first preprocessing module in the embodiment of the present invention is used to perform word segmentation processing on the input query Q and document D of the model according to the word segmentation rules corresponding to the first LLM model to obtain the corresponding first word segmentation sequence and second word segmentation sequence; and by adding a preset separator word 'SEP' between the first and second word segmentation sequences, perform sequence connection on the first and second word segmentation sequences to obtain the corresponding third word segmentation sequence = the first word segmentation sequence ⊕ 'SEP' ⊕ the second word segmentation sequence, where ⊕ is a string concatenation symbol; and perform embedding encoding processing on the third word segmentation sequence according to the embedding encoding rules of the first LLM model to obtain the corresponding embedding vector X{x i}; and use the vector length of the embedding vector X{x i} as the initial vector length and the output vector feature dimension of the first decoder of the first LLM model as the initial vector feature dimension, and set a all-zero vector as the corresponding initial decoding vector Y0{y 0,i}; and combine the embedding vector X{x i} and the initial decoding vector Y0{y 0,i}Send to the first LLM model;

[0101] Here, the embedding vector X{x i} is composed of multiple embedding sub-vectors x i Sorted in sequence, 1 ≤ sub-vector index i ≤ N1, where N1 is the total number of word segments in the third word segment sequence, and the embedding sub-vector x i Corresponds one-to-one with the word segments of the third word segment sequence; Initialize the decoding vector Y0{y 0,i} includes N1 initial decoding sub-vectors y 0,i , and the initial decoding sub-vector y 0,i Corresponds one-to-one with the embedding sub-vector x i One-to-one;

[0102] 2) The first LLM model:

[0103] When the first LLM model of the embodiment of the present invention is used for the document scoring model, its first encoder is used to perform high-dimensional feature encoding on the embedding vector X{x i} to obtain the corresponding encoded vector E{e i} and send it to the first decoder; The first decoder is used to use the initial decoding vector Y0{y 0,i} as the initial state quantity of the decoding vector and perform multiple loop iterations of decoding according to the encoded vector E{e i} to obtain the corresponding decoding vector Y{y i} and send it to the first MLP model;

[0104] Here, the vector length and vector feature dimension of the decoding vector Y{y i} are the same as those of the initial decoding vector Y0{y 0,i}, and the decoding vector Y{y i} includes N1 decoding sub-vectors y i , and the decoding sub-vector y i Corresponds one-to-one with the initial decoding sub-vector y 0,i One-to-one;

[0105] The first MLP model:

[0106] The first MLP model of the embodiment of the present invention is composed of a first input layer, a first hidden layer, and a first output layer connected in sequence; The first input layer is composed of a single fully connected layer; The first hidden layer is composed of one or more first fully connected activation layers, and each first fully connected activation layer is composed of a single fully connected layer and a ReLU activation function layer connected in sequence; The first output layer is composed of a single fully connected layer and a Sigmoid activation function layer connected in sequence; The output data of the first output layer is a probability data output by the Sigmoid activation function layer;

[0107] Generally speaking, the first MLP model is used to convert the decoding vector Y{yi} Enter the first input layer, and through the layer-by-layer operations of the first input layer, the first hidden layer, and the first output layer, obtain the probability data output by the Sigmoid activation function layer, and use the probability data obtained this time as the corresponding score S output; here, the value range of the score S in the embodiments of the present invention is between 0 and 1.

[0108] As Figure 2 shown, the first model input end of the document ranking model is used to receive the query Q, the second model input end is used to receive the document sequence {D j}, and the model output end is used to output the corresponding ranking sequence {Id j}. The internal model components of this document ranking model include: a second preprocessing module, a first LLM model, a segmentation mapping module, a second MLP model, and a ranking output module.

[0109] The connection relationship of the internal components of the document ranking model is: the first and second input ends of the second preprocessing module are connected to the first and second model input ends of the document ranking model, the first output end is connected to the input end of the first LLM model, and the second output end is connected to the first input end of the segmentation mapping module; the output end of the first LLM model is connected to the second input end of the segmentation mapping module; the output end of the segmentation mapping module is connected to the input end of the second MLP model; the output end of the second MLP model is connected to the input end of the ranking output module; the output end of the ranking output module is connected to the model output end of the document ranking model.

[0110] The functions of the internal components of the document ranking model are as follows:

[0111] 1) The second preprocessing module:

[0112] The second preprocessing module in the embodiments of the present invention is used to count the total number of documents in the document sequence {D j} and use the statistical result as the corresponding total number of documents N D ; and perform word segmentation on the input query Q according to the word segmentation rules corresponding to the first LLM model to obtain the corresponding fourth word segmentation sequence; and perform word segmentation on each document D in the document sequence {D j} to obtain the corresponding fifth word segmentation sequence; and perform sequence connection on the fourth word segmentation sequence and each fifth word segmentation sequence through the separator word segmentation 'SEP' to obtain the corresponding sixth word segmentation sequence = the fourth word segmentation sequence ⊕ 'SEP' ⊕ the fifth word segmentation sequence; and perform word segmentation on N j sixth word segmentation sequences through the preset line break word segmentation '\n' according to the document D D and jThe seventh token sequence is obtained by sequentially connecting the sorting order of [[ID=]], which is equal to the sixth token sequence ⊕ '\n' ⊕ the sixth token sequence ⊕... ⊕ '\n' ⊕ the sixth token sequence; and the seventh token sequence is processed by embedding encoding according to the embedding encoding rule of the first LLM model to obtain the corresponding embedding vector X{x k}; and using the vector length of the embedding vector X{x k} as the initial vector length and the output vector feature dimension of the first decoder of the first LLM model as the initial vector feature dimension, a all-zero vector is set as the corresponding initial decoding vector Y0{y 0,k}; and the sub-vector index k corresponding to each line break token '\n' in the embedding vector X{x k} is recorded as the corresponding segmentation mark position, and all the obtained segmentation mark positions form the corresponding position set G; and the embedding vector X{x k} and the initial decoding vector Y0{y 0,k} are sent to the first LLM model; and the total number of documents N D and the position set G are sent to the segmentation mapping module;

[0113] Here, the embedding vector X{x k} is sequentially sorted by multiple embedding sub-vectors x k , 1 ≤ sub-vector index k ≤ N2, N2 is the total number of tokens in the seventh token sequence, and the embedding sub-vector x k corresponds to the tokens in the seventh token sequence one by one; the initial decoding vector Y0{y 0,k} includes N2 initial decoding sub-vectors y 0,k , and the initial decoding sub-vector y 0,k corresponds to the embedding sub-vector x k one by one;

[0114] 2) The first LLM model:

[0115] When the first LLM model of the embodiment of the present invention is used for the document sorting model, its processing process is similar to its processing process in the document scoring model. Specifically: the first encoder is used to perform high-dimensional feature encoding on the embedding vector X{x k} to obtain the corresponding encoding vector E{e k} and send it to the first decoder; the first decoder is used to perform multiple loop iterations of decoding according to the encoding vector E{e 0,k} with the initial decoding vector Y0{y k} as the initial state quantity of the decoding vector to obtain the corresponding decoding vector Y{y k} and send it to the segmentation mapping module;

[0116] Here, the decoding vector Y{y k} has the same vector length and vector feature dimension as the initial decoded vector Y0{y 0,k}, and the decoded vector Y{y k} includes N2 decoded sub-vectors y k , and the decoded sub-vector y k corresponds one-to-one with the initial decoded sub-vector y 0,k ;

[0117] 3) Piecewise mapping module:

[0118] The piecewise mapping module of the embodiment of the present invention is used to sequentially divide the decoded vector Y{y D} into N k fragment decoded vectors according to the total number of documents N D and all the segmented marker positions of the position set G and pad each variable-length fragment decoded vector by zero-padding into a fixed-length fragment decoded vector with a vector length of a preset vector length L and, with the input vector feature space of the second input layer of the second MLP model as the target feature space, perform vector projection of each fixed-length fragment decoded vector in the target feature space to obtain a fixed-length fragment projection vector h j ; and form a corresponding projection vector H{h D} composed of N j fragment projection vectors h j and send it to the second MLP model;

[0119] Here, the fragment decoded vector corresponds one-to-one with the sixth word segmentation sequence in the seventh word segmentation sequence; the vector length L is a preset positive integer;

[0120] 4) Second MLP model:

[0121] The second MLP model of the embodiment of the present invention is composed of a second input layer, a second hidden layer, and a second output layer connected; the second input layer is composed of a single fully-connected layer; the second hidden layer is composed of one or more second fully-connected activation layers, and each second fully-connected activation layer is composed of a single fully-connected layer and a ReLU activation function connected; the second output layer is composed of a single fully-connected layer and a Softmax function layer; the fully-connected layer of the second output layer outputs a process vector with a vector length of N D and a vector feature dimension of N D as well, and the Softmax function layer of the second output layer is used to regard the current process vector as an N D ×N D two-dimensional matrix and perform N DThe probability distribution of the rankings is calculated to obtain a D The probability sub-vector p composed of the ranking probability data j , and the obtained N D probability vector p j Form a corresponding probability vector P{p j};

[0122] In general, the second MLP model is used to transform the projection vector H{h j} input into the second input layer, and the probability vector P{p j}Send to the sorting output module;

[0123] Here, the probability vector P{p j}Include N D probability vector p j ; Probability subvector p j With Document D j One-to-one correspondence; probability subvector p j By N D The ranking probability data is composed of rank probability data, and the ranking probability data corresponds to the sorting rank c one by one;

[0124] 5) Sorting output module:

[0125] The sorting output module of the embodiment of the present invention is used to convert the probability vector P{p j} into an N D ×N D A two-dimensional matrix A, each row of the two-dimensional matrix A corresponds to a probability sub-vector p j , each column corresponds to a ranking rank c, each matrix unit matches the ranking probability data corresponding to the current row and current column; and the document index j corresponding to the matrix unit with the largest probability value in each column of the two-dimensional matrix A is used as the document number Id corresponding to the current column c ; and sorted by c from 1 to N D The order of the obtained N D Document ID c Sort to get the corresponding sort sequence {Id c} and output.

[0126] Step 2: Under the condition that the model parameters of the first LLM model remain unchanged, the document scoring model and the document ranking model are trained separately based on the preset scoring dataset and ranking dataset;

[0127] Specifically comprising: step 21, training a first MLP model of a document scoring model based on a scoring dataset;

[0128] Among them, the scoring dataset includes a first training query set, a first training document set, and a first label set. The first training query set includes multiple first training queries, the first training document set includes multiple first training documents, and the first label set includes multiple first relevance labels. The value of the first relevance label is 0 or 1. Each first relevance label is used to mark the relevance of any group (first training query, first training document). A value of 0 indicates irrelevance, and a value of 1 indicates relevance.

[0129] Specifically, it includes: Step 211, count the total number of the first training queries in the scoring dataset to obtain the corresponding total number M1, and count the total number of the first training documents to obtain the corresponding total number M2; and denote each first training query as the corresponding query q u and each first training document as the corresponding document d v and each first relevance label as the corresponding label score and form M1×M2 first data groups from M1 queries q u , M2 documents d v and M1×M2 label scores

[0130] Step 212, use the query q and document d u of each first data group v as the corresponding query Q and document D to input into the similarity scoring model for relevance scoring prediction, and take the score S obtained from this prediction as the corresponding predicted score and form a corresponding first prediction-label pair from the predicted scores corresponding to each first data group and label scores

[0131] Step 213, input the obtained M1×M2 first prediction-label pairs into the preset first model loss function L A ; and based on the preset first model optimizer, modulate the model parameters of the first MLP model of the document scoring model in the direction of minimizing the first model loss function L A for one round.

[0132] Here, the first model loss function L of the embodiment of the present invention A is implemented based on the binary cross-entropy loss function, specifically:

[0133] ​​

[0134] The first model optimizer in the embodiments of the present invention includes at least an Adam optimizer and an SGD optimizer;

[0135] Step 214. At the end of this round of modulation, then each first data group (q u , d v , a u,v )'s query q u and document d v are used as the corresponding query Q and document D and input into the similarity scoring model for correlation scoring prediction, and the score S obtained from this prediction is used as the corresponding predicted score And from each first data group the corresponding predicted score and the label score form a corresponding second prediction-label pair And the obtained M1×M2 second prediction-label pairs are brought into the preset first model evaluation function F A for calculation to obtain the corresponding first evaluation value;

[0136] Here, the first model evaluation function F in the embodiments of the present invention A is implemented based on the RMSE function, specifically:

[0137]

[0138] Step 215. Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to Step 212 to continue training; if so, confirm that the model training of the document scoring model ends;

[0139] Here, the first evaluation value range in the embodiments of the present invention is a preset numerical range;

[0140] Step 22. And based on the sorted data set, train the segment mapping module and the second MLP model of the document sorting model;

[0141] Among them, the sorted data set includes multiple first data records; the first data record includes a second training query, a second training document sequence, and a first label sequence; the second training document sequence is sorted by multiple second training documents, and each second training document corresponds to a sequence index value; the first label sequence is composed of multiple first label numbers, the total number of first label numbers matches the total number of second training documents, and the number value of each first label number corresponds to the sequence index value of a second training document;

[0142] Specifically, it includes: Step 221, splitting the sorted data set into two sub-data sets according to a preset first splitting ratio, denoted as the corresponding first training set and first evaluation set;

[0143] Here, the first splitting ratio in the embodiment of the present invention is a preset ratio parameter, such as 8:2; both the first training set and the first evaluation set are composed of multiple first data records; the ratio of the total number of records in the first training set and the first evaluation set satisfies the first splitting ratio;

[0144] Step 222, recording the total number of records in the first training set as M3; and taking each first data record in the first training set as the corresponding current training record; and denoting the second training query of the current training record as the corresponding query q r , 1 ≤ index r ≤ M3; and counting the total number of documents in the second training document sequence of the current training record to obtain the corresponding total number M r,4 , and denoting the current second training document sequence as the corresponding document sequence {d r,w}, 1 ≤ index w ≤ M r,4 ; and denoting the first label sequence of the current training record as the corresponding current label sorting sequence; and setting a label vector with both the vector length and the vector feature dimension being M r,4 ;

[0145] Among them, the document sequence {d r,w} is composed of the corresponding M r,4 documents d r,w ;

[0146] The label vector is composed of the corresponding M r,4 label sub-vectors ; the label sub-vector corresponds to the document d r,w one by one; the label sub-vector is composed of the corresponding M r,4 label probability data , 1 ≤ index g ≤ M r,4 , and each label probability data corresponds to a ranking in the current label sorting sequence; among the label sub-vectors only one label probability data is 1, and the remaining M r,4 - 1 label probability data are all 0, and the label probability data that is 1 corresponds to the ranking of the document d r,w corresponding to the current label sub-vector in the current label sorting sequence;

[0147] Step 223: Take each query q in the first training set r and its corresponding document sequence {d r,w} as the corresponding query Q and document sequence {D j} and input them into the document ranking model for ranking; and during this processing, take the probability vector P{p j} output by the second MLP model of the document ranking model as the corresponding prediction vector and form a corresponding third prediction-label pair with each label vector in the first training set and its corresponding prediction vector where the prediction vector

[0148] is composed of corresponding M prediction sub-vectors r,4 ; the prediction sub-vector corresponds one-to-one with the document d ; the prediction sub-vector r,w is composed of corresponding M prediction probability data r,4

[0149] Step 224: Substitute the obtained M3 third prediction-label pairs into the preset second model loss function L B ; and based on the preset second model optimizer, perform one round of modulation on the model parameters of the piecewise mapping module and the second MLP model of the document ranking model in the direction of minimizing the second model loss function L B Here, the first model loss function L

[0150]

[0151] A of the embodiment of the present invention is implemented based on the multi-class cross-entropy loss function, specifically:

[0151]

[0152] The second model optimizer of the embodiment of the present invention includes at least the Adam optimizer and the SGD optimizer;

[0153] Step 225: Denote the total number of records in the first evaluation set as M5; and take each first data record in the first evaluation set as the corresponding current evaluation record; and denote the second training query of the current evaluation record as the corresponding query q l l,6 where 1 ≤ index l ≤ M5; and count the total number of documents in the second training document sequence of the current evaluation record to obtain the corresponding total number M l,s and denote the current second training document sequence as the corresponding document sequence {d l,s}, 1 ≤ index s ≤ M l,6 ; and denote the first tag sequence of the current evaluation record as the corresponding current tag sorting sequence; and set a tag vector with both the vector length and the vector feature dimension being M l,6

[0154] where the document sequence {d l,s} consists of the corresponding M l,6 documents d l,s ;

[0155] The tag vector consists of the corresponding M l,6 tag sub-vectors ; The tag sub-vector corresponds one-to-one with the document d l,s ; The tag sub-vector consists of the corresponding M l,6 tag probability data , 1 ≤ index t ≤ M l,6 , and each tag probability data corresponds to a sorting rank in the current tag sorting sequence; only one tag probability data in each tag sub-vector is 1, and the remaining M l,6 -1 tag probability data are all 0, and the tag probability data that is 1 corresponds to the sorting rank of the document d l,s corresponding to the current tag sub-vector in the current tag sorting sequence;

[0156] Step 226, use each query q l in the first evaluation set and its corresponding document sequence {d l,s} as the corresponding query Q and document sequence {D j} and input them into the document sorting model for sorting; and use the probability vector P{p j} output by the second MLP model of the document sorting model during this processing as the corresponding prediction vector And form a corresponding fourth prediction-tag pair from each tag vector in the first evaluation set and its corresponding prediction vector

[0157] where the prediction vector consists of the corresponding M l,6 prediction sub-vectors ; The prediction sub-vector corresponds to the document d l,s ​​One-to-one correspondence; predicted sub-vector by the corresponding M l,6 prediction probability data constitute;

[0158] Step 227, and bring the obtained M5 fourth prediction-label pairs into the preset second model evaluation function F B for calculation to obtain the corresponding second evaluation value;

[0159] Here, the second model evaluation function F of the embodiment of the present invention B is implemented based on the RMSE function, specifically:

[0160]

[0161] Step 228, identify whether the second evaluation value meets the preset second evaluation value range; if not, return to Step 222 to continue training; if so, confirm that the model training of the document ranking model ends;

[0162] Here, the second evaluation value range of the embodiment of the present invention is a preset numerical range;

[0163] Step 23, and confirm that the training of both models ends after the training of the document scoring model and the document ranking model ends.

[0164] Step 3, after the training of both models ends, receive the query text input by the user as the corresponding first query text; and use the document library specified by the current user as the corresponding first document library.

[0165] Here, the first document library of the embodiment of the present invention includes multiple first documents.

[0166] Step 4, perform a first-stage document screening on the first document library according to the first query text in the word frequency scoring manner to obtain the corresponding initial screening document sequence;

[0167] Specifically include: Step 41, perform keyword extraction processing on the first query text to obtain the corresponding keyword set; and record each first document in the first document library as the corresponding document D z ;

[0168] Among them, the keyword set consists of one or more keywords kw m constitute, 1 ≤ index m ≤ N KEY , N KEY is the total number of keywords in the keyword set; 1 ≤ index z ≤ N Z , N Z is the total number of documents in the first document library;

[0169] Step 42: Based on the keyword set and each document D z calculate the corresponding first-stage relevance score sm z ;

[0170] Here, the first-stage relevance score sm of the embodiment of the present invention z is calculated as follows:

[0171]

[0172] wherein, TF(kw m , D z ) is the word frequency of the current keyword kw m in the current document D z ; IDF(kw m ) is the inverse document frequency of the current keyword kw m in the first document library; k1 and b are two preset adjustment parameters; |D z | is the document length of the current document D z ; avgdl is the average document length of all the first documents in the first document library; n(kw m ) is the total number of the first documents containing the current keyword kw m in the first document library;

[0173] Step 43: Sort the first documents in the first document library with the first-stage relevance score sm z higher than the preset first scoring threshold in descending order of the first-stage relevance score sm z to obtain the corresponding preliminary screening document sequence.

[0174] Here, the first scoring threshold is a preset threshold parameter.

[0175] Step 5: Use the document scoring model to perform second-stage document screening on the preliminary screening document sequence according to the first query text to obtain the corresponding secondary screening document sequence;

[0176] Specifically, it includes: taking the first query text and each first document in the preliminary screening document sequence as the corresponding query Q and document D and inputting them into the document scoring model for correlation scoring prediction, and taking the score S output by the current prediction as the corresponding second-stage relevance score; and sorting the first documents in the preliminary screening document sequence with the second-stage relevance score higher than the preset second scoring threshold in descending order of the second-stage relevance score to obtain the corresponding secondary screening document sequence.

[0177] Here, the second scoring threshold is a preset threshold parameter.

[0178] Step 6, use the document ranking model to perform three-stage document re-ranking on the second-screened document sequence according to the first query text to obtain the corresponding recommended document sequence and feedback it to the current user;

[0179] Specifically, it includes: taking the first query text and the second-screened document sequence as the corresponding query Q and document sequence {D j} and inputting them into the document ranking model for ranking, and taking the ranking sequence {Id c} output by the current ranking as the corresponding current ranking sequence; and re-ranking all the first documents in the second-screened document sequence according to the document number sorting order of the current ranking sequence to obtain the corresponding recommended document sequence and feedback it to the current user.

[0180] Figure 3 FIG. 11 is a module structure diagram of a retrieval and ranking device based on a large language model provided in the second embodiment of the present invention. The device is a terminal device or a server for implementing the foregoing method embodiment, or may be a device that enables the foregoing terminal device or server to implement the foregoing method embodiment. For example, the device may be a device or a chip system of the foregoing terminal device or server. As Figure 3 shown, the device includes: a model construction module 201, a model training module 202, a retrieval information receiving module 203, a first-stage retrieval and ranking module 204, a second-stage retrieval and ranking module 205, and a third-stage retrieval and ranking module 206.

[0181] The model construction module 201 is used to select a large language model based on the Transformer framework with an encoder-decoder structure and that has been pre-trained as the first LLM model; and construct two task models by configuring a downstream MLP model on the output side of the first LLM model, denoted as a document scoring model and a document ranking model; the document scoring model is used to predict the relevance score between the input query Q and document D of the model and output the corresponding score S; the document ranking model is used to rank the input document sequence {D j} according to the input query Q of the model and output the corresponding ranking sequence {Id c}; the document sequence {D j} is composed of multiple documents D j sorted, 1 ≤ document index j ≤ N D where N D is the total number of documents in the document sequence {D j}; the ranking sequence {Id c} is composed of N D document numbers Id c where 1 ≤ ranking position c ≤ N D and each document number Id c corresponds to a document index j; the ranking sequence {Id cThe higher the ranking in the sorting, the higher the matching degree between the corresponding document and the query.

[0182] The model training module 202 is used to separately train the document scoring model and the document sorting model based on a preset scoring data set and a sorting data set under the condition that the model parameters of the first LLM model remain unchanged.

[0183] The retrieval information receiving module 203 is used to, after the training of both models is completed, receive the query text input by the user as the corresponding first query text; and use the document library specified by the current user as the corresponding first document library; the first document library includes a plurality of first documents.

[0184] The first-stage retrieval and sorting module 204 is used to perform a first-stage document screening on the first document library according to the first query text in the word frequency scoring manner to obtain the corresponding pre-screened document sequence.

[0185] The second-stage retrieval and sorting module 205 is used to perform a second-stage document screening on the pre-screened document sequence according to the first query text by using the document scoring model to obtain the corresponding sub-screened document sequence.

[0186] The third-stage retrieval and sorting module 206 is used to perform a third-stage document re-sorting on the sub-screened document sequence according to the first query text by using the document sorting model to obtain the corresponding recommended document sequence and feedback it to the current user.

[0187] A retrieval and sorting device based on a large language model provided by an embodiment of the present invention can execute the method steps in the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here.

[0188] It should be noted that it should be understood that the division of each module of the above device is only a logical function division. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by a processing element; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the model construction module can be a separately established processing element, or can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a certain processing element of the above device to perform the functions of the above determined modules. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or can be independently implemented. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit in the processor element or the instruction in the form of software.

[0189] For example, the above-mentioned modules may be one or more integrated circuits configured to implement the above method, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), etc. Again, when a certain module above is implemented in the form of a processing element scheduler code, the processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processors that can call program code. Again, these modules may be integrated together and implemented in the form of a System-on-a-chip (SOC).

[0190] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the foregoing method embodiments are generated in whole or in part. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.). The above computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The above available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0191] Figure 4 The structural schematic diagram of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server that implements the method of the foregoing embodiments, or may be a terminal device or a server that is connected to the foregoing terminal device or server and implements the method of the foregoing embodiments. As Figure 4As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver operations of the transceiver 303. Various instructions may be stored in the memory 302 for completing various processing functions and implementing the processing steps described in the method of the foregoing embodiments. Preferably, the electronic device according to the embodiment of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to implement communication connections between components. The above-mentioned communication port 306 is used for the electronic device to connect and communicate with other peripherals.

[0192] The Figure 4 system bus 305 mentioned in may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The system bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to implement communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include a Random Access Memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0193] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), a Graphics Processing Unit (GPU), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0194] It should be noted that the embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored, and when the instructions are run on a computer, the computer is caused to execute the methods and processing procedures provided in the above embodiments.

[0195] An embodiment of the present invention provides a retrieval and ranking method, apparatus, electronic device, and computer-readable storage medium based on a large language model. As can be seen from the above, the embodiment of the present invention selects a large language model that is implemented based on an encoder-decoder structure of the Transformer framework and has been pre-trained as the first LLM model; and constructs two task models (document scoring model, document ranking model) by configuring a downstream MLP model on the output side of the first LLM model; and trains the two task models separately under the condition that the model parameters of the first LLM model remain unchanged; and uses the two task models to improve the retrieval accuracy after the training is completed. Specifically: first, perform a first-stage document screening on the document library specified by the user according to the query text input by the user in the word frequency scoring manner to obtain a preliminary screening document sequence, then use the document scoring model to perform a second-stage document screening on the preliminary screening document sequence again according to the query text to obtain a secondary screening document sequence, and finally use the document ranking model to perform a third-stage document re-ranking on the secondary screening document sequence according to the query text to obtain a corresponding recommended document sequence and feedback it to the current user. When processing the retrieval and ranking task, the embodiment of the present invention first performs a full-library relevant document retrieval and ranking on the target library through a conventional ranking method, then performs a secondary document screening and ranking on the preliminary screening document sequence generated by the conventional ranking method through the document scoring model, and finally performs a re-ranking on the secondary screening result, that is, the secondary screening document sequence, again according to the relevance level through the document ranking model. The embodiment of the present invention achieves the effect of converging the retrieval document range, improves the relevance recognition accuracy between the document and the query, and enhances the retrieval accuracy of the document-level retrieval task.

[0196] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented in hardware, software modules executed by a processor, or a combination of both. The software modules may be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0197] The specific embodiments described above have further elaborated on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above description is only the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A retrieval and ranking method based on large language models, characterized in that, The method includes: A large language model based on the Transformer framework and pre-trained in the encoder-decoder structure is selected as the first LLM model; and two task models are constructed by configuring the downstream MLP model on the output side of the first LLM model, which are recorded as the document scoring model and the document ranking model; the document scoring model is used to predict the relevance score of the query Q and document D input to the model and output the corresponding score S; the document ranking model is used to sort the document sequence {D j } sort and output the corresponding sort sequence {Id c }; the document sequence {D j }D by multiple documents j Sorted, 1≤document indexj≤N D , N D For the document sequence {D j }; the sorted sequence {Id c } by N D Document ID c Composition, 1≤rank c≤N D , each of the document numbers Id c Corresponding to a document index j; the sorting sequence {Id c The higher the ranking in the}, the higher the matching degree between the corresponding document and the query; Under the condition that the model parameters of the first LLM model remain unchanged, the document scoring model and the document ranking model are respectively trained based on a preset scoring dataset and a ranking dataset; After the training of both models is completed, the query text input by the user is received as the corresponding first query text; and the document library specified by the current user is used as the corresponding first document library; the first document library includes multiple first documents; According to the word frequency scoring method, the first document library is screened at the first stage according to the first query text to obtain the corresponding initial screening document sequence; The document scoring model is used to screen the initial screening document sequence at the second stage according to the first query text to obtain the corresponding secondary screening document sequence; The document ranking model is used to re-rank the secondary screening document sequence at the third stage according to the first query text to obtain the corresponding recommended document sequence and feedback it to the current user.

2. The retrieval and ranking method based on a large language model according to claim 1, wherein The first LLM model is composed of a first encoder and a first decoder connected; the first encoder is implemented based on the Encoder model structure of the Transformer framework, and the first decoder is implemented based on the Decoder model structure of the Transformer framework; The first model input end of the document scoring model is used to receive the query Q, the second model input end is used to receive the document D, and the model output end is used to output the corresponding score S; the document scoring model includes a first preprocessing module, the first LLM model and a first MLP model; the first and second input ends of the first preprocessing module are connected to the first and second model input ends of the document scoring model, and the output end is connected to the input end of the first LLM model; the output end of the first LLM model is connected to the input end of the first MLP model; The output end of the first MLP model is connected to the model output end of the document scoring model; The first model input end of the document sorting model is used to receive the query Q, and the second model input end is used to receive the document sequence {D j}, and the model output end is used to output the corresponding sorting sequence {Id j}; the document sorting model includes a second preprocessing module, the first LLM model, a segmentation mapping module, a second MLP model, and a sorting output module; the first and second input ends of the second preprocessing module are connected to the first and second model input ends of the document sorting model, the first output end is connected to the input end of the first LLM model, and the second output end is connected to the first input end of the segmentation mapping module; the output end of the first LLM model is connected to the second input end of the segmentation mapping module; the output end of the segmentation mapping module is connected to the input end of the second MLP model; the output end of the second MLP model is connected to the input end of the sorting output module; the output end of the sorting output module is connected to the model output end of the document sorting model; The scoring dataset includes a first training query set, a first training document set and a first label set. The first training query set includes multiple first training queries, the first training document set includes multiple first training documents, and the first label set includes multiple first relevance labels; The value of the first relevance label is 0 or 1; each first relevance label is used to mark the relevance of any group (first training query, first training document). A value of 0 indicates irrelevance, and a value of 1 indicates relevance; The ranking dataset includes multiple first data records; the first data record includes a second training query, a second training document sequence and a first label sequence; the second training document sequence is sorted by multiple second training documents, and each second training document corresponds to a sequence index value; the first label sequence is composed of multiple first label numbers, the total number of the first label numbers matches the total number of the second training documents, and the number value of each first label number corresponds to the sequence index value of a second training document.

3. The retrieval and ranking method based on a large language model according to claim 2, wherein The first preprocessing module is used to perform word segmentation processing on the query Q and the document D input to the model according to the word segmentation rules corresponding to the first LLM model to obtain corresponding first word segmentation sequences and second word segmentation sequences; and by adding a preset separator word 'SEP' between the first and second word segmentation sequences, the first and second word segmentation sequences are concatenated to obtain a corresponding third word segmentation sequence = the first word segmentation sequence ⊕ 'SEP' ⊕ the second word segmentation sequence, where ⊕ is a string concatenation symbol; and perform embedding encoding processing on the third word segmentation sequence according to the embedding encoding rules of the first LLM model to obtain a corresponding embedding vector X{x i}; and use the vector length of the embedding vector X{x i} as the initial vector length and the output vector feature dimension of the first decoder of the first LLM model as the initial vector feature dimension, and set a vector of all zeros as the corresponding initial decoding vector Y0{y 0,i}; and send the embedding vector X{x i} and the initial decoding vector Y0{y 0,i} to the first LLM model; the embedding vector X{x i} is composed of multiple embedded sub-vectors x i sorted in order, 1 ≤ sub-vector index i ≤ N1, where N1 is the total number of word segments in the third word segmentation sequence, and the embedded sub-vector x i corresponds one-to-one with the word segments of the third word segmentation sequence; the initial decoding vector Y0{y 0,i} includes N1 initial decoding sub-vectors y 0,i , and the initial decoding sub-vector y 0,i corresponds one-to-one with the embedded sub-vector x i ; The first encoder of the first LLM model is used to perform high-dimensional feature encoding on the embedding vector X{x i} to obtain the corresponding encoded vector E{e i} and send it to the first decoder; the first decoder of the first LLM model is used to use the initial decoding vector Y0{y 0,i} as the initial state quantity of the decoding vector and perform multiple loop iterations of decoding according to the encoded vector E{e i} to obtain the corresponding decoding vector Y{y i} and send it to the first MLP model; the vector length and vector feature dimension of the decoding vector Y{y i} are consistent with those of the initial decoding vector Y0{y 0,i}, the decoding vector Y{y i} includes N1 decoding sub-vectors y i ,the decoding sub-vectors y i and the initial decoding sub-vectors y 0,i are in one-to-one correspondence; the first MLP model is formed by connecting a first input layer, a first hidden layer, and a first output layer; the first input layer is composed of a single fully-connected layer; the first hidden layer is composed of one or more first fully-connected activation layers, and each of the first fully-connected activation layers is formed by connecting a fully-connected layer and a ReLU activation function layer; the first output layer is formed by connecting a fully-connected layer and a Sigmoid activation function layer; the output data of the first output layer is a probability data output by the Sigmoid activation function layer; The first MLP model is used to input the decoded vector Y{y i} into the first input layer, and through the layer-by-layer operations of the first input layer, the first hidden layer, and the first output layer, obtain the probability data output by the Sigmoid activation function layer, and use the probability data obtained in that iteration as the corresponding score S output; the value range of the score S is between 0 and 1.

4. The retrieval and ranking method based on a large language model according to claim 2, wherein The second preprocessing module is used to count the total number of documents in the document sequence {D j}, and use the statistical result as the corresponding total number of documents N D ; and perform word segmentation on the query Q input to the model according to the word segmentation rule corresponding to the first LLM model to obtain the corresponding fourth word segmentation sequence; and perform word segmentation on each document D in the document sequence {D j} according to the word segmentation rule corresponding to the first LLM model to obtain the corresponding fifth word segmentation sequence; and perform sequence connection on the fourth word segmentation sequence and each of the fifth word segmentation sequences through the separator word segmentation 'SEP' to obtain the corresponding sixth word segmentation sequence = fourth word segmentation sequence j 'SEP' 'SEP' fifth word segmentation sequence; and perform sequential connection on N D of the sixth word segmentation sequences according to the sorting order of the document D j through the preset line break word segmentation '\n' to obtain the corresponding seventh word segmentation sequence = sixth word segmentation sequence '\n' sixth word segmentation sequence …… '\n' sixth word segmentation sequence; and perform embedding encoding processing on the seventh word segmentation sequence according to the embedding encoding rule of the first LLM model to obtain the corresponding embedding vector X{x k}; and use the vector length of the embedding vector X{x k} as the initial vector length and the output vector feature dimension of the first decoder of the first LLM model as the initial vector feature dimension, and set a all-zero vector as the corresponding initial decoding vector Y0{y 0,k}; and record the sub-vector index k corresponding to each line break word segmentation '\n' in the embedding vector X{x k} as the corresponding segmentation mark position, and form the corresponding position set G from all the obtained segmentation mark positions; and send the embedding vector X{x k} and the initial decoding vector Y0{y 0,k} to the first LLM model; and send the total number of documents N D and the position set G to the segmentation mapping module; the embedding vector X{x k} is composed of multiple embedding sub-vectors x k sorted in sequence, 1 ≤ sub-vector index k ≤ N2, N2 is the total number of word segments of the seventh word segmentation sequence, and the embedding sub-vector x k corresponds one-to-one with the word segments of the seventh word segmentation sequence; the initial decoding vector Y0{y 0,k} including N2 initial decoding sub-vectors y 0,k , the initial decoding sub-vector y 0,k corresponds one-to-one with the embedding sub-vector x k ; The first encoder of the first LLM model is used to perform high-dimensional feature encoding on the embedding vector X{x k} to obtain the corresponding encoded vector E{e k} and send it to the first decoder; the first decoder of the first LLM model is used to use the initial decoding vector Y0{y 0,k} as the initial state quantity of the decoding vector and perform multiple cyclic iterative decodings according to the encoded vector E{e k} to obtain the corresponding decoding vector Y{y k} and send it to the segmentation mapping module; the vector length and vector feature dimension of the decoding vector Y{y k} are consistent with those of the initial decoding vector Y0{y 0,k}, the decoding vector Y{y k} includes N2 decoding sub-vectors y k and the decoding sub-vectors y k correspond one-to-one with the initial decoding sub-vectors y 0,k ; The segment mapping module is used to calculate the total number of documents N D and all the segment marker positions of the position set G will be the decoded vector Y{y k }Sequentially split into N D fragment decoding vectors And each variable length segment decoding vector is converted into Fill in a fixed-length segment decoding vector with a vector length of the preset vector length L The input vector feature space of the second input layer of the second MLP model is used as the target feature space, and the decoding vectors of each fixed-length segment are decoded based on a built-in MLP model. Perform vector projection of the target feature space to obtain a fixed-length segment projection vector h j ; and by N D The fragment projection vector h j The corresponding projection vector H{h j } is sent to the second MLP model; the segment decoding vector One-to-one correspondence with the sixth participle sequence in the seventh participle sequence; The second MLP model is formed by connecting the second input layer, the second hidden layer, and the second output layer; the second input layer consists of a single fully-connected layer; the second hidden layer consists of one or more second fully-connected activation layers, and each second fully-connected activation layer is formed by connecting a fully-connected layer and a ReLU activation function; the second output layer consists of a fully-connected layer and a Softmax function layer; the fully-connected layer of the second output layer outputs a process vector with a vector length of N D and a vector feature dimension of also N D The Softmax function layer of the second output layer is used to treat the current process vector as an N D ×N D two-dimensional matrix and calculate the probability distribution of N D ranking positions for each row of this two-dimensional matrix to obtain a probability sub-vector p D consisting of N j ranking probability data, and N D such probability sub-vectors p j are used to form a corresponding probability vector P{p j}; The second MLP model is used to input the projection vector H{h j} into the second input layer, and through the layer-by-layer operations of the second input layer, the second hidden layer, and the second output layer, obtain the probability vector P{p j} output by the Softmax function layer and send it to the sorting output module; The probability vector P{p j} includes N D probability sub-vectors p j ; the probability sub-vector p j corresponds to the document D j one by one; the probability sub-vector p j is composed of N D ranking probability data, and the ranking probability data corresponds to the sorting ranking c one by one; The sorting and output module is used to convert the probability vector P{p j} into a two-dimensional matrix A of N D ×N D . Each row of the two-dimensional matrix A corresponds to one of the probability sub-vectors p j , each column corresponds to one of the sorting ranks c, and each matrix element matches the rank probability data corresponding to the current row and the current column; and the document index j corresponding to the matrix element with the largest probability value in each column of the two-dimensional matrix A is used as the document number Id corresponding to the current column c ; and the N D obtained in the order of the sorting rank c from 1 to N D document numbers Id c are sorted to obtain the corresponding sorting sequence {Id c} and output.

5. The retrieval and ranking method based on a large language model according to claim 2, wherein separately training the document scoring model and the document ranking model based on a preset scoring data set and a ranking data set specifically includes: training the first MLP model of the document scoring model based on the scoring data set; training the segment mapping module and the second MLP model of the document ranking model based on the ranking data set; and confirming that the training of both models is completed after the training of the document scoring model and the document ranking model is completed.

6. The retrieval and ranking method based on a large language model according to claim 5, wherein The training of the first MLP model of the document scoring model based on the scoring data set specifically includes: Step 61, count the total number of the first training queries in the scoring data set to obtain the corresponding total number M1, and count the total number of the first training documents to obtain the corresponding total number M2; and denote each of the first training queries as the corresponding query q u 、denote each of the first training documents as the corresponding document d v 、denote each of the first relevance labels as the corresponding label score and from M1 of the queries q u 、M2 of the documents d v and M1×M2 of the label scores form M1×M2 first data groups Step 62, for each of the first data groups query q u and document d v as the corresponding query Q and document D, input them into the similarity scoring model for correlation score prediction, and use the score S obtained from this prediction as the corresponding predicted score and form a corresponding first prediction-label pair with the corresponding predicted score and the label score for each of the first data groups Step 63: Bring the obtained M1×M2 first prediction-label pairs into a preset first model loss function L A ; and based on a preset first model optimizer, perform one round of modulation on the model parameters of the first MLP model of the document scoring model in the direction of minimizing the first model loss function L A to reach the minimum value; Among them, the first model loss function L A is implemented based on the binary cross-entropy loss function, specifically: the first model optimizer includes at least an Adam optimizer and an SGD optimizer; Step 64, at the end of this round of modulation, then use the query q u , d v , a u,v ) of each of the first data groups (q u and the document d v as the corresponding query Q and document D and input them into the similarity scoring model for correlation scoring prediction, and use the score S obtained from this prediction as the corresponding prediction score and use the corresponding prediction scores of each of the first data groups and the label scores to form a corresponding second prediction-label pair and use the obtained M1×M2 second prediction-label pairs to calculate the corresponding first evaluation value by substituting them into the preset first model evaluation function F A ; Among them, the first model evaluation function F A is implemented based on the RMSE function, specifically: Step 65, identifying whether the first evaluation value meets a preset first evaluation value range; if not, returning to step 62 to continue training; if so, confirming that the model training of the document scoring model is completed.

7. The retrieval and ranking method based on the large language model according to claim 5, wherein The training of the segment mapping module and the second MLP model of the document ranking model based on the ranking data set specifically includes: Step 71, splitting the ranking data set into two sub-data sets denoted as corresponding first training set and first evaluation set based on a preset first splitting ratio; wherein, both the first training set and the first evaluation set are composed of multiple first data records; the ratio of the total number of records in the first training set and the first evaluation set meets the first splitting ratio; Step 72, record the total number of records in the first training set as M3; and take each of the first data records in the first training set as the corresponding current training record; and denote the second training query of the current training record as the corresponding query q r , where 1 ≤ index r ≤ M3; and count the total number of documents in the second training document sequence of the current training record to obtain the corresponding total M r,4 , and denote the current second training document sequence as the corresponding document sequence {d r,w}, where 1 ≤ index w ≤ M r,4 ; and denote the first label sequence of the current training record as the corresponding current label sorting sequence; and set a label vector with both the vector length and the vector feature dimension being M based on the current label sorting sequence r,4 ​ Among them, the document sequence {d r,w} is composed of the corresponding M r,4 documents d r,w ; The label vector is composed of corresponding M r,4 label sub-vectors ; the label sub-vectors correspond one-to-one with the document d r,w ; the label sub-vectors are composed of corresponding M r,4 label probability data , where 1 ≤ index g ≤ M r,4 , and each of the label probability data corresponds to a ranking in the current label sorting sequence; only one of the label probability data in each of the label sub-vectors is 1, and the remaining M r,4 -1 label probability data are all 0, and the label probability data that is 1 corresponds to the ranking of the document d corresponding to the current label sub-vector r,w in the current label sorting sequence; Step 73, for each query q in the first training set r and its corresponding document sequence {d r,w}, use them as the corresponding query Q and document sequence {D j} and input them into the document ranking model for ranking; and during this processing, use the probability vector P{p j} output by the second MLP model of the document ranking model as the corresponding prediction vector and form a corresponding third prediction-label pair from each label vector in the first training set and its corresponding prediction vector ​ Among them, the prediction vector is composed of corresponding M r,4 prediction sub-vectors ; the prediction sub-vectors correspond one-to-one with the document d r,w ; the prediction sub-vectors are composed of corresponding M r,4 prediction probability data . Step 74, bring the obtained M3 third prediction-label pairs into a preset second model loss function L B ; and based on a preset second model optimizer, perform one round of modulation on the model parameters of the segment mapping module of the document sorting model and the second MLP model in the direction of minimizing the second model loss function L B ; Among them, the first model loss function L A is implemented based on the multi-class cross-entropy loss function, specifically: the second model optimizer includes at least an Adam optimizer and an SGD optimizer; Step 75, record the total number of records in the first evaluation set as M5; and use each of the first data records in the first evaluation set as the corresponding current evaluation record; and denote the second training query of the current evaluation record as the corresponding query q l , where 1 ≤ index l ≤ M5; and count the total number of documents in the second training document sequence of the current evaluation record to obtain the corresponding total number M l,6 , and denote the current second training document sequence as the corresponding document sequence {d l,s}, where 1 ≤ index s ≤ M l,6 ; and denote the first label sequence of the current evaluation record as the corresponding current label sorting sequence; and set a label vector with both the vector length and the vector feature dimension being M based on the current label sorting sequence l,6 ​ Among them, the document sequence {d l,s} is composed of the corresponding M l,6 documents d l,s ; The label vector is composed of corresponding M l,6 label sub-vectors ; the label sub-vectors correspond one-to-one with the document d l,s ; the label sub-vectors are composed of corresponding M l,6 label probability data , 1 ≤ index t ≤ M l,6 , and each of the label probability data corresponds to a ranking position in the current label sorting sequence; only one of the label probability data in each of the label sub-vectors is 1, and the remaining M l,6 - 1 label probability data are all 0, and the label probability data that is 1 corresponds to the ranking position of the document d l,s corresponding to the current label sub-vector in the current label sorting sequence; Step 76, each query q in the first evaluation set l and its corresponding document sequence {d l,s} are used as the corresponding query Q and document sequence {D j} and input into the document ranking model for ranking processing; and during this processing, the probability vector P{p j} output by the second MLP model of the document ranking model is used as the corresponding prediction vector and a corresponding fourth prediction-label pair is formed by each label vector in the first evaluation set and its corresponding prediction vector ​ Among them, the prediction vector is composed of corresponding M l,6 prediction sub-vectors ; the prediction sub-vectors correspond one-to-one with the document d l,s ; the prediction sub-vectors are composed of corresponding M l,6 prediction probability data . Step 77, and bring the obtained M5 fourth prediction-label pairs into a preset second model evaluation function F B for calculation to obtain the corresponding second evaluation value; Among them, the second model evaluation function F B is implemented based on the RMSE function, specifically: Step 78, identifying whether the second evaluation value meets a preset second evaluation value range; if not, returning to step 72 to continue training; if so, confirming that the model training of the document ranking model is completed.

8. The retrieval and ranking method based on a large language model according to claim 1, wherein The one-stage document screening of the first document library according to the first query text by the word frequency scoring method specifically includes: Step 81, perform keyword extraction processing on the first query text to obtain a corresponding keyword set; and denote each of the first documents in the first document library as a corresponding document D z ; Among them, the keyword set consists of one or more keywords kw m constituting, where 1 ≤ index m ≤ N KEY , N KEY being the total number of keywords in the keyword set; 1 ≤ index z ≤ N Z , N Z being the total number of documents in the first document library; Step 82, based on the keyword set and each of the documents D z calculate the corresponding first-stage relevance score sm z ; Among them, the one-stage correlation score sm z is calculated as follows: TF(kw m , D z ) is the term frequency of the current keyword kw m in the current document D z ; IDF(kw m ) is the inverse document frequency of the current keyword kw m in the first document library; k1 and b are two preset adjustment parameters; |D z | is the document length of the current document D z ; avgdl is the average document length of all the first documents in the first document library; n(kw m ) is the total number of the first documents in the first document library that contain the current keyword kw m . Step 83: From the first-stage relevance score sm in the first document library z The first documents with a first-stage relevance score sm higher than a preset first scoring threshold are sorted in descending order according to the first-stage relevance score sm z to obtain the corresponding pre-screened document sequence.

9. The retrieval and ranking method based on a large language model according to claim 1, characterized in that The two-stage document screening of the initial screening document sequence according to the first query text by using the document scoring model specifically includes: Input the first query text and each of the first documents in the initially screened document sequence as the corresponding query Q and document D into the document scoring model for correlation scoring prediction, and use the score S output by the current prediction as the corresponding two-stage correlation score; and sort the first documents in the initially screened document sequence whose two-stage correlation scores are higher than a preset second scoring threshold in descending order of the two-stage correlation scores to obtain the corresponding secondarily screened document sequence.

10. The retrieval and sorting method based on the large language model according to claim 1, characterized in that Using the document ranking model to perform three-stage document re-ranking on the secondarily screened document sequence according to the first query text to obtain the corresponding recommended document sequence and feedback it to the current user, specifically including: Take the first query text and the secondary screening document sequence as the corresponding query Q and document sequence {D j} and input them into the document ranking model for ranking, and take the ranking sequence {Id c} output by the current ranking as the corresponding current ranking sequence; and re-rank all the first documents in the secondary screening document sequence according to the document number sorting order of the current ranking sequence to obtain the corresponding recommended document sequence and feedback it to the current user.

11. An apparatus for performing the retrieval and ranking method based on a large language model according to any one of claims 1-10, characterized in that, The apparatus includes: a model construction module, a model training module, a retrieval information receiving module, a first-stage retrieval ranking module, a second-stage retrieval ranking module, and a third-stage retrieval ranking module; The model construction module is used to select a large language model based on the Transformer framework's encoder-decoder structure that has been pre-trained as the first LLM model; and two task models are constructed by configuring a downstream MLP model on the output side of the first LLM model, denoted as the document scoring model and the document ranking model; the document scoring model is used to predict the relevance score between the query Q and the document D input to the model and output the corresponding score S; the document ranking model is used to rank the document sequence {D j} input to the model according to the query Q input to the model and output the corresponding ranking sequence {Id c}; the document sequence {D j} is composed of multiple documents D j sorted, 1 ≤ document index j ≤ N D , N D being the total number of documents in the document sequence {D j}; the ranking sequence {Id c} is composed of N D document numbers Id c s, 1 ≤ ranking position c ≤ N D , each document number Id c corresponding to a document index j; the higher the ranking position in the ranking sequence {Id c}, the higher the matching degree between the corresponding document and the query. The model training module is used to separately train the document scoring model and the document ranking model based on a preset scoring data set and ranking data set under the condition that the model parameters of the first LLM model remain unchanged; The retrieval information receiving module is used to receive the query text input by the user as the corresponding first query text after both model trainings are completed; and use the document library specified by the current user as the corresponding first document library; the first document library includes multiple first documents; The first-stage retrieval ranking module is used to perform first-stage document screening on the first document library according to the first query text in a word frequency scoring manner to obtain the corresponding initially screened document sequence; The second-stage retrieval ranking module is used to perform second-stage document screening on the initially screened document sequence using the document scoring model according to the first query text to obtain the corresponding secondarily screened document sequence; The third-stage retrieval ranking module is used to perform three-stage document re-ranking on the secondarily screened document sequence using the document ranking model according to the first query text to obtain the corresponding recommended document sequence and feedback it to the current user.

12. An electronic device, characterized in that, Including: A memory, a processor, and a transceiver; The processor is used to be coupled with the memory, read and execute the instructions in the memory to implement the method according to any one of claims 1-10; The transceiver is coupled with the processor, and the processor controls the transceiver to perform message sending and receiving.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1-10.

Images