A dense retrieval method and device combining a large language model and a pseudo query technology

Abstract

The embodiment of the application relates to a dense retrieval method and device combining a large language model and a pseudo query technology, the method comprising: selecting a first large model; and constructing a pseudo query-document scoring model with a BERT model as a first and a second encoder; extracting a query-feedback seed sample; configuring three large language model instruction templates; constructing a first data set according to the seed sample and the first and second instruction templates, and training the pseudo query-document scoring model based on the data set; after the training is completed, receiving a query text and historical retrieval information input by a user side, extracting a query-feedback text pair based on the historical retrieval information, generating pseudo feedback based on the extracted text pair sequence and the query text, and composing a pseudo query from the query and the pseudo feedback, and performing dense retrieval based on the pseudo query to obtain a retrieval result and feed back the retrieval result to the current user. The application can improve retrieval accuracy and reduce the probability of retrieval ambiguity problems.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a dense retrieval method and device combining a large language model and a pseudo query technology. BACKGROUND

[0002] In the application field of big data and artificial intelligence technology, how to accurately obtain the required information is one of the core problems that information retrieval technology needs to solve. Dense retrieval is a kind of mainstream retrieval technology at present. In this technology, the query object (referred to as query) and the information document (referred to as document) are respectively converted into dense semantic vectors in a high-dimensional feature space, and the retrieval results (also referred to as feedback) are filtered based on the vector similarity between the query and the document. Because dense retrieval can obtain deep semantic features from a high-dimensional feature space, the purpose of improving retrieval accuracy can be achieved. However, we have found a problem in practical application: if the query text content is short, even if it can be converted into a dense semantic vector in a high-dimensional feature space, the personalized semantic features obtained are not clear, which may cause the retrieval results to not match the retrieval intent, that is, the retrieval ambiguity problem. For example, the user inputs the query text as "cloud", which is related to at least the fields of meteorology, computer and communication. Without knowing the current user's field of interest, even if the dense retrieval technology is used, the retrieval ambiguity problem cannot be avoided.

[0003] Pseudo query is a query expansion technology. The principle is to synthesize a pseudo feedback text related to the expected feedback corresponding to the real query in advance, and then splice the real query and the pseudo feedback to form a new pseudo query, and input the pseudo query into the retriever for retrieval. In principle, if the semantic relevance between the synthesized pseudo feedback and the expected real feedback is high, more explicit personalized semantic features can be added to the pseudo query, which can naturally improve the retrieval accuracy and reduce the probability of the occurrence of the retrieval ambiguity problem. However, if there is no substantial correlation between the synthesized pseudo feedback and the expected real feedback, not only can effective personalized semantic features not be added, but more feature interference / noise can be brought. In this case, the probability of the occurrence of the retrieval ambiguity problem is likely to increase, and the retrieval accuracy is likely to decrease. That is, if you want to introduce pseudo query into dense retrieval to improve retrieval accuracy and reduce the probability of the occurrence of the retrieval ambiguity problem, a solution must be given to solve the relevance problem between the pseudo feedback and the expected feedback.

[0004] Large Language Model (LLM) is a task processing system realized based on deep learning technology. The Natural Language Processing (NLP) capability of the LLM model is very powerful: 1) can learn and remember a series of individualized features such as the user's search habits, knowledge field of interest, etc. with the user's historical search information as the context; 2) can extract a formatted query-feedback text pair from the user's historical search information; 3) can generate a segment of relevant reply text for the new question (i.e. query) specified by the user at this time based on the user input question and answer task description and question and answer sample. That is to say, based on the LLM model, a segment of reply text related to the user's expected feedback, i.e. pseudo-feedback text, can be generated under the premise of knowing the user's historical search information.

[0005] Based on the above analysis, we conclude that if the LLM model, pseudo query technology and dense search technology can be combined, the probability of search ambiguity can be more reliably reduced. How to combine the LLM model, pseudo query technology and dense search technology has become a technical problem that needs to be solved by the present application. SUMMARY

[0006] The present application aims at the defects of the prior art, and provides a dense retrieval method and device combining a large language model and a pseudo query technology, an electronic device, and a computer readable storage medium. The present application pre-selects a generative large language model that has completed pre-training and NLP task fine-tuning as a first large model, selects a BERT series model that has completed pre-training as a first and second encoder, constructs a pseudo query-document scoring model with the first and second encoders as the core, and takes a preset information document library as a target document library. Then, a specified number N1 of target documents are randomly extracted from each knowledge field of the target document library as seed feedback, a corresponding query text is set for each seed feedback, denoted as seed query, and each seed query and the corresponding seed feedback form a corresponding query-feedback seed sample. Then, a large language model instruction template for generating a corresponding feedback when the given sample set is taken as a reference and the given query is not empty, and generating a new query-feedback text pair when the given query is empty, denoted as the first instruction template, a second instruction template for generating a specified number N2 of negative feedbacks for the given positive feedback, and requiring that the text feature similarity between the positive and negative feedbacks is higher than a preset similarity threshold a but the semantic correlation is lower than a preset correlation threshold b, and a third instruction template for extracting query-feedback text pairs with the given retrieval information as the current context. Then, a model training data set is constructed for the pseudo query-document scoring model according to N1 x M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model, and the pseudo query-document scoring model is trained based on the data set. After the model training is completed, the first query text and the first historical retrieval information input by the user side are received, the query-feedback text pair extraction is performed based on the first historical retrieval information, the third instruction template, and the first large model to obtain a first text pair sequence, the pseudo feedback generation processing is performed based on the first text pair sequence, the first query text, the first instruction template, and the first large model to obtain a first pseudo feedback text, the first query text and the first pseudo feedback text form a first pseudo query text, and the dense retrieval is performed based on the first pseudo query text, the pseudo query-document scoring model, and the target document library to obtain a first retrieval result, which is fed back to the current user. The present application combines the LLM model, the pseudo query technology, and the dense retrieval technology, which not only can further improve the retrieval accuracy, but also can reduce the probability of occurrence of retrieval ambiguity problems.

[0007] To achieve the above object, a first aspect of an embodiment of the present application provides a dense retrieval method combining a large language model and a pseudo query technology, which comprises:

[0008] selecting a generative large language model that has completed pre-training and NLP task fine-tuning as a first large model; selecting a BERT series model that has completed pre-training as a corresponding first encoder and a second encoder, and constructing a pseudo query-document scoring model with the first and second encoders as cores; and selecting a preset information document library as a corresponding target document library, and counting the total number of knowledge fields of the target document library to obtain a corresponding total number M; the pseudo query-document scoring model is used for relevance scoring processing of a pseudo query Q and a document D input by the model and outputs a corresponding score S, the value range of the score S is between -1 and 1; the target document library includes a plurality of target documents; each target document corresponds to a knowledge field; and the total number M is a positive integer;

[0009] randomly selecting a specified number N1 of target documents from each knowledge field of the target document library as N1 seed feedbacks; setting a corresponding query text for each seed feedback, denoted as a seed query; and forming a corresponding query-feedback seed sample from each seed query and the corresponding seed feedback; the specified number N1 is a positive integer; and the total number of query-feedback seed samples is N1XM;

[0010] configuring a large language model instruction template for generating a corresponding feedback when a given query is not empty and generating a new query-feedback text pair when the given query is empty, denoted as a first instruction template; configuring a second instruction template for generating a specified number N2 of negative feedbacks for a given positive feedback, and requiring that the text feature similarity between the positive and negative feedbacks is higher than a preset similarity threshold a but the semantic correlation is lower than a preset correlation threshold b; and configuring a third instruction template for query-feedback text pair extraction with a given search information as the current context; the specified number N2 is a positive integer; the value range of the text feature similarity and the similarity threshold a is between 0 and 1; the value range of the semantic correlation and the correlation threshold b is between -1 and 1, and the smaller the value of the semantic correlation is, the weaker the correlation is when the semantic correlation is positive, and the existence of semantic contradiction is indicated when the semantic correlation is negative;

[0011] constructing a model training data set for the pseudo query-document scoring model according to N1XM query-feedback seed samples, the first instruction template, the second instruction template, and the first large model, denoted as a first data set;

[0012] model training the pseudo query-document scoring model based on the first data set;

[0013] After model training, the system receives the first query text and the first historical retrieval information input by the user; it then performs query-feedback text pair extraction processing based on the first historical retrieval information, the third instruction template, and the first large model to obtain the corresponding first text pair sequence; it then performs pseudo-feedback generation processing based on the first text pair sequence, the first query text, the first instruction template, and the first large model to obtain the corresponding first pseudo-feedback text; the first query text and the first pseudo-feedback text together form the corresponding first pseudo-query text; and it performs dense retrieval processing based on the first pseudo-query text, the pseudo-query-document scoring model, and the target document library to obtain the corresponding first retrieval result; and it then feeds back the first retrieval result to the current user; the first historical retrieval information consists of all retrieval information of the current user within the most recent specified time period.

[0014] Preferably, the generative large language model is any one of the following: GPT series models, BERT series models, T5 series models, LLaMA series models, Wenxin series models, or Qwen series models.

[0015] Preferably, the first model input terminal of the pseudo-query-document rating model is used to receive the pseudo-query Q input by the model, the second model input terminal is used to receive the document D input by the model, and the model output terminal is used to output the corresponding rating S;

[0016] The pseudo-query-document scoring model includes a first preprocessing module, a second preprocessing module, a first encoder, a second encoder, and a scoring module. The input of the first preprocessing module is connected to the input of the first model, and its output is connected to the input of the first encoder. The input of the second preprocessing module is connected to the input of the second model, and its output is connected to the input of the second encoder. The output of the first encoder is connected to the first input of the scoring module. The output of the second encoder is connected to the second input of the scoring module. The output of the scoring module is connected to the output of the model.

[0017] The first preprocessing module is used to remove stop words, special characters / symbols, and punctuation marks from the pseudo-query Q to obtain the corresponding first preprocessed text; and to segment the first preprocessed text according to the segmentation rules of the BERT model to obtain the corresponding first segmented sequence; and to add a preset classification tag CLS to the beginning of the first segmented sequence to obtain the corresponding second segmented sequence; and to perform embedding encoding according to the embedding encoding rules of the BERT model based on the second segmented sequence to obtain the corresponding embedding encoding vector E. QSend to the first encoder; the total number of words in the first word segmentation sequence is denoted as the total number of words w1, and the total number of words in the second word segmentation sequence is denoted as the total number of words w2, where w2 = 1 + w1; the embedded encoding vector E Q Composed of w2 sub-encoded vectors e Q,i Composition, 1 ≤ vector index i ≤ w2;

[0018] The second preprocessing module is used to perform sentence segmentation on the document D to obtain multiple corresponding first sentences A. j And for the first sentence A j The total number of sentences is counted to obtain the corresponding total number of sentences N3, where 1 ≤ sentence index j ≤ N3; and each of the first sentences A j The second preprocessed text B is obtained by removing stop words, special characters / symbols, and punctuation marks. j ; and according to the word segmentation rules of the BERT model, each of the second preprocessed texts B j The corresponding third-order word sequence C is obtained by word segmentation. j ; and in the first of the aforementioned third word segmentation sequences C j＝1 The classification marker CLS is added to the head; and in the 2nd to N3rd third word segmentation sequence C j A preset delimiter marker SEP is added to the beginning of the sequence; and the third word segmentation sequence C with N3 completed markers is added. j The corresponding fourth word segment sequence is obtained by sequentially concatenating the segments; and the corresponding embedding encoding vector E is obtained by embedding the fourth word segment sequence according to the embedding encoding rules of the BERT model. D Send to the second encoder; the total number of segments in the fourth segmentation sequence is denoted as the total number of segments w3; the embedding encoding vector E D Composed of w3 sub-encoded vectors e D,k Composition, 1 ≤ vector index k ≤ w3;

[0019] The first encoder is used to process the input embedding encoding vector E Q High-dimensional semantic feature encoding is performed to obtain the corresponding encoding tensor H. Q Send to the scoring module; the encoded tensor H Q Composed of w2 sub-encoding vectors h Q,i Composition; the first sub-encoding vector h Q,i＝1 Corresponding to the classification marker CLS of the second word segmentation sequence;

[0020] The second encoder is used to process the input embedding encoding vector E D High-dimensional semantic feature encoding is performed to obtain the corresponding encoding tensor H. D Send to the scoring module; the encoded tensor H DComposed of w3 sub-encoding vectors h D,k Composition; the sub-encoding vector h D,k The feature dimension and the sub-encoding vector h Q,i The feature dimensions are consistent; the first sub-encoding vector h D,k＝1 It corresponds to the classification tag CLS of the fourth word segmentation sequence;

[0021] The scoring module is used to process the encoded tensor H. Q The first sub-encoding vector h Q,i＝1 Extract it as the corresponding first vector, and then use the encoded tensor H D The first sub-encoding vector h D,k＝1 The first vector is extracted and used as the corresponding second vector; the similarity between the first and second vectors is calculated based on the cosine similarity algorithm to obtain the corresponding first similarity; and the first similarity is output as the corresponding score S.

[0022] Preferably, the configurable parameters of the first instruction template include the given sample set and the given query; the given sample set consists of multiple first samples; the first samples consist of a first sample query and a first sample feedback.

[0023] The first instruction template is a formatted text consisting of multiple lines. The first line is formatted as: "Task:" + first task description text, which prompts the first large model to generate corresponding feedback for the given query based on the given sample set. The second to second-to-last lines correspond one-to-one with the first sample in the given sample set, and each line is formatted as: "Sample x:" + "Query:" + first sample query + first separator + "Feedback:" + first sample feedback + second separator, where x is the sample number starting from 1 and incrementing by 1. The "+" in the line format is a character sequence concatenation character. The first separator defaults to a semicolon ";", and the second separator defaults to a period ".". The last line is formatted as: "Query:" + given query.

[0024] When the given query of the first instruction template is empty, the first large model will output a corresponding query-feedback text pair according to the sample format; when the given query of the first instruction template is not empty, the first large model will only output the corresponding feedback text.

[0025] The configurable parameters of the second instruction template include the given positive feedback;

[0026] The second instruction template is a formatted text consisting of four lines. The first line is formatted as: "Task:" + second task description text, which prompts the first large model to generate N2 negative feedbacks with similar text features but weak semantic correlation or semantic contradiction for the given positive feedback. The second line is formatted as: "Requirement 1:" + first requirement text, which requires that the text feature similarity between each negative feedback generated by the first large model and the given positive feedback should be higher than the similarity threshold a, and specifies that the text feature similarity value ranges from 0 to 1. The third line is formatted as: "Requirement 2:" + second requirement text, which requires that the semantic correlation between each negative feedback generated by the first large model and the given positive feedback should be lower than the correlation threshold b, and specifies that the semantic correlation value ranges from -1 to 1, and indicates that the semantic correlation is positive when there is a weak correlation and negative when there is semantic contradiction. The fourth line is formatted as: "Positive feedback:" + the given positive feedback.

[0027] The configurable parameters of the third instruction template include the given search information;

[0028] The third instruction template is a formatted text consisting of two paragraphs; the first line of text is formatted as: "Task:" + third task description text, which is used to prompt the first large model to extract query-feedback text pairs in the context of the given search information; the second line of text is formatted as: "Search information:" + given search information;

[0029] The first dataset includes multiple first data records; the first data records include a first training query, a first positive feedback, and a first negative feedback sequence; the first negative feedback sequence consists of N2 first negative feedbacks; the first positive feedback and each of the first negative feedbacks have similar textual features but are weakly semantically correlated or have semantic contradictions.

[0030] Preferably, the step of constructing a model training dataset, denoted as the first dataset, for the pseudo-query-document scoring model based on N1×M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model specifically includes:

[0031] Step 51: Set a first counter initialized to 0;

[0032] Step 52: A corresponding first instruction is obtained by setting the given sample set of the first instruction template to the corresponding N1×M query-feedback seed samples and setting the given query to empty; the first instruction is input into the first large model to generate query-feedback text pairs to obtain the corresponding first query-feedback text pairs; the query text of the first query-feedback text pair is used as a corresponding first training query, and the feedback text of the first query-feedback text pair is used as a corresponding first positive feedback.

[0033] The first query-feedback text pair consists of a query text and a feedback text.

[0034] Step 53: Obtain a corresponding second instruction by setting the given positive feedback of the second instruction template as the current first positive feedback; input the second instruction into the first large model for negative feedback generation processing to obtain N2 negative feedbacks that are similar to the current first positive feedback text features but are weakly related or semantically contradictory; and take each negative feedback obtained this time as a corresponding first negative feedback, and form the corresponding first negative feedback sequence from the N2 first negative feedbacks obtained this time.

[0035] Step 54: The current first training query and its corresponding first positive feedback and first negative feedback sequences form a corresponding first data record;

[0036] Step 55: Increment the first counter by 1; and identify whether the incremented first counter exceeds a preset threshold for the total number of records; if it does not exceed the threshold, return to step 52; if it does exceed the threshold, then the first dataset is composed of all the first data records obtained.

[0037] Preferably, the step of training the pseudo-query-document rating model based on the first dataset specifically includes:

[0038] Step 61: Based on a preset first segmentation ratio, the first dataset is divided into two sub-datasets, denoted as the first training set and the first evaluation set.

[0039] Wherein, both the first training set and the first evaluation set are composed of multiple first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0040] Step 62: Calculate a minimum loss l based on the preset temperature parameter τ and the specified quantity N2. min ;

[0041] Wherein, the minimum loss lmin The calculation formula is:

[0042]

[0043] The minimum loss l min In the calculation formula, + represents a plus sign;

[0044] Step 63: Extract the first data record of the first training set as the corresponding current training record;

[0045] Step 64: Input the first training query and the first positive feedback from the current training record as the corresponding pseudo-query Q and document D into the pseudo-query-document scoring model for relevance scoring processing, and use the score S output by the model in this processing as the corresponding positive sample score s. p The first training query and each of the first negative feedbacks in the first negative feedback sequence of the current training record are used as the corresponding pseudo-query Q and document D, respectively, and input into the pseudo-query-document scoring model for correlation scoring processing. The score S output by the model in this processing is used as a corresponding negative sample score s. n,g 1 ≤ negative sample index g ≤ N2;

[0046] Step 65: Set the maximum value of the positive sample score to 1; and set the obtained positive sample score s... p and N2 negative sample scores s n,g Substitute the preset loss function L; and based on the preset first model optimizer, optimize the model towards improving the positive sample score s. p The positive sample score reaches its maximum value, and the loss function L reaches its minimum loss l. min In the direction of the pseudo-query-document scoring model, the model parameters of the first and second encoders are fine-tuned in one round;

[0047] Wherein, the loss function L is:

[0048]

[0049] In the loss function L, + stands for plus sign;

[0050] The first model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0051] Step 66: Identify whether the current training record is the last first data record of the first training set; if yes, proceed to step 67; if no, take the next first data record of the first training set as the new current training record and return to step 64.

[0052] Step 67, set the minimum loss l min As the corresponding first label loss; and perform a round of traversal on all the first data records of the first evaluation set; and during this round of traversal, take the currently traversed first data record as the corresponding current evaluation record; and take the first training query and the first positive feedback of the current evaluation record as the corresponding pseudo query Q and the document D as input to the pseudo query-document scoring model for relevance scoring processing, and take the score S output by the model in this processing as the corresponding positive sample score s. p The first training query and each of the first negative feedbacks in the first negative feedback sequence of the current evaluation record are used as the corresponding pseudo-query Q and document D, respectively, and input into the pseudo-query-document scoring model for correlation scoring processing. The score S output by the model in this processing is used as a corresponding negative sample score s. n,g ; and the positive sample scores s obtained this time p and N2 negative sample scores s n,g The loss function L is substituted into the calculation and the calculation result is used as a corresponding first prediction loss; the current first prediction loss and the first label loss are combined into a corresponding first prediction-label pair; and at the end of this round of traversal, all the obtained first prediction-label pairs are substituted into the preset model evaluation function to calculate the corresponding first evaluation value.

[0053] The model evaluation function is implemented based on the MAE function, MSE function, or RMSE function.

[0054] Step 68: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 63 to continue training; if it meets the range, stop training and confirm that the model training is complete.

[0055] Preferably, the step of extracting the corresponding first text pair sequence based on the first historical retrieval information, the third instruction template, and the first large model specifically includes:

[0056] A corresponding third instruction is obtained by setting the given retrieval information of the third instruction template to the current first historical retrieval information; the third instruction is input into the first large model to extract query-feedback text pairs to obtain one or more corresponding query-feedback text pairs; the query text and feedback text of each extracted query-feedback text pair are combined to form a corresponding first text pair; and all the obtained first text pairs are combined to form a corresponding first text pair sequence.

[0057] Preferably, the step of generating the corresponding first pseudo-feedback text based on the first text pair sequence, the first query text, the first instruction template, and the first large model specifically includes:

[0058] A corresponding fourth instruction is obtained by setting the given sample set of the first instruction template as the current first text pair sequence and setting the given query as the current first query text; the fourth instruction is then input into the first large model for feedback text generation processing, and the feedback text generated by the model in this processing is used as the corresponding first pseudo feedback text.

[0059] Preferably, the first pseudo-query text, composed of the first query text and the first pseudo-feedback text, specifically includes:

[0060] The first query text is used as the corresponding first string, the first pseudo feedback text is used as the corresponding second string, and the third string, which is formed by sequentially concatenating the first string, the preset delimiter mark SEP, and the second string, is used as the corresponding first pseudo query text.

[0061] Preferably, the step of performing dense retrieval processing based on the first pseudo-query text, the pseudo-query-document rating model, and the target document library to obtain the corresponding first retrieval result specifically includes:

[0062] Each target document in the target document library is taken as the corresponding current document, and the first pseudo-query text and the current document are taken as the corresponding pseudo-query Q and document D, respectively. The pseudo-query-document rating model is then used for relevance rating processing, and the rating S output by the model in this processing is taken as a corresponding first document rating. The highest first document rating is taken as the corresponding maximum rating. One or more target documents in the target document library that correspond to the maximum rating are extracted to form the corresponding first retrieval result.

[0063] A second aspect of the present invention provides an apparatus for implementing the dense retrieval method combining large language model and pseudo-query technology as described in the first aspect above. The apparatus includes: a model preparation module, a seed sample preparation module, an instruction template preparation module, a dataset preparation module, a scoring model training module, and a dense retrieval module.

[0064] The model preparation module selects a pre-trained and NLP task-fine-tuned generative large language model as the first large model; selects a pre-trained BERT series model as the corresponding first encoder and second encoder, and constructs a pseudo-query-document scoring model based on the first and second encoders; and uses a preset information document library as the corresponding target document library, and calculates the total number of knowledge domains in the target document library to obtain the corresponding total number M; the pseudo-query-document scoring model performs correlation scoring on the pseudo-query Q and document D input to the model and outputs the corresponding score S, where the score S ranges from -1 to 1; the target document library includes multiple target documents; each target document corresponds to a knowledge domain; and the total number M is a positive integer;

[0065] The seed sample preparation module is used to randomly select a specified number N1 target documents from each knowledge domain of the target document library as corresponding N1 seed responses; and to set a corresponding query text for each seed response, denoted as a seed query; and to form a corresponding query-feedback seed sample by each seed query and the corresponding seed response; the specified number N1 is a positive integer; the total number of query-feedback seed samples is N1×M;

[0066] The instruction template preparation module is used to configure a large language model instruction template, denoted as the first instruction template, for generating a corresponding feedback when a given query is not empty, and generating a new query-feedback text pair when the given query is empty, with reference to a given sample set; and to configure a second instruction template for generating a specified number N2 negative feedbacks for a given positive feedback, requiring that the text feature similarity between the positive and negative feedbacks be higher than a preset similarity threshold a but lower than a preset relevance threshold b; and to configure a third instruction template for extracting query-feedback text pairs with the given retrieval information as the current context; the specified number N2 is a positive integer; the text feature similarity and the similarity threshold a range from 0 to 1; the semantic relevance and the relevance threshold b range from -1 to 1, where a smaller positive value indicates a weaker relevance, and a negative value indicates a semantic contradiction;

[0067] The dataset preparation module is used to construct a model training dataset for the pseudo-query-document scoring model based on N1×M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model, denoted as the first dataset;

[0068] The scoring model training module is used to train the pseudo-query-document scoring model based on the first dataset;

[0069] The dense retrieval module is used to receive a first query text and a first historical retrieval information input by the user after model training is completed; and to perform query-feedback text pair extraction processing based on the first historical retrieval information, the third instruction template, and the first large model to obtain a corresponding first text pair sequence; and to perform pseudo-feedback generation processing based on the first text pair sequence, the first query text, the first instruction template, and the first large model to obtain a corresponding first pseudo-feedback text; and to form a corresponding first pseudo-query text by combining the first query text and the first pseudo-feedback text; and to perform dense retrieval processing based on the first pseudo-query text, the pseudo-query-document scoring model, and the target document library to obtain a corresponding first retrieval result; and to feed back the first retrieval result to the current user; the first historical retrieval information consists of all retrieval information of the current user within the most recent specified time period.

[0070] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0071] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0072] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0073] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0074] This invention provides a dense retrieval method, apparatus, electronic device, and computer-readable storage medium that combines a large language model and pseudo-query technology. As described above, this invention pre-selects a pre-trained generative large language model with NLP task fine-tuning as the first large model, and a pre-trained BERT series model as the first and second encoders. A pseudo-query-document scoring model is constructed using the first and second encoders as the core, and a pre-defined information document library is used as the target document library. A specified number of N1 target documents are randomly selected from each knowledge domain of the target document library as seed feedback, and a corresponding query text is set for each seed feedback, denoted as a seed query. Each seed query and its corresponding seed feedback constitute a corresponding query-feedback seed sample. A large language model instruction template is configured to generate a corresponding feedback when the given query is not empty, and a new query-feedback text pair when the given query is empty, denoted as the first instruction template. A specified number of N2 negative feedbacks are configured for a given positive feedback, requiring the text feature similarity between the positive and negative feedbacks to be higher than a preset similarity threshold. A second instruction template with a value of 'a' but semantic relevance lower than a preset relevance threshold 'b' is used. A third instruction template is configured to extract query-feedback text pairs using given retrieval information as the current context. A model training dataset is then constructed for the pseudo-query-document scoring model based on N1×M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model. The pseudo-query-document scoring model is trained based on this dataset. After model training, the first query text and first historical retrieval information input from the user are received. A first text pair sequence is obtained by extracting query-feedback text pairs based on the first historical retrieval information, the third instruction template, and the first large model. A first pseudo-feedback text is generated based on the first text pair sequence, the first query text, the first instruction template, and the first large model. The first pseudo-query text is composed of the first query text and the first pseudo-feedback text. A dense retrieval is performed based on the first pseudo-query text, the pseudo-query-document scoring model, and the target document library to obtain the first retrieval result, which is then fed back to the current user. This embodiment of the invention combines the LLM model, pseudo-query technology, and dense retrieval technology, which not only improves retrieval accuracy but also reduces the probability of retrieval ambiguity. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of a dense retrieval method combining a large language model and pseudo-query technology provided in Embodiment 1 of the present invention;

[0076] Figure 2 This is a module structure diagram of the pseudo-query-document scoring model provided in Embodiment 1 of the present invention;

[0077] Figure 3 A schematic diagram of the template format of the first instruction template, the second instruction template, and the third instruction template provided in Embodiment 1 of the present invention;

[0078] Figure 4 This is a module structure diagram of a dense retrieval device combining a large language model and pseudo-query technology provided in Embodiment 2 of the present invention;

[0079] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0081] Embodiment 1 of the present invention provides a dense retrieval method that combines a large language model and pseudo-query technology, such as... Figure 1 The diagram illustrates a dense retrieval method combining a large language model and pseudo-query technology, as provided in Embodiment 1 of the present invention. This method mainly includes the following steps:

[0082] Step 1: Select a pre-trained generative large language model that has been fine-tuned for NLP tasks as the first large model; select a pre-trained BERT series model as the corresponding first encoder and second encoder, and build a pseudo-query-document scoring model with the first and second encoders as the core; and use a preset information document library as the corresponding target document library, and count the total number of knowledge domains in the target document library to obtain the corresponding total number M.

[0083] Here, the generative large language model in this embodiment of the invention is any one of the GPT series models, BERT series models, T5 series models, LLaMA series models, Wenxin series models, or Qwen series models. The target document library in this embodiment of the invention includes multiple target documents; each target document corresponds to a knowledge domain; the total number M of knowledge domains in the target document library is a positive integer.

[0084] The pseudo-query-document scoring model in this embodiment of the invention is used to perform correlation scoring on the pseudo-query Q and document D input to the model and output the corresponding score S, where the score S ranges from -1 to 1. For example... Figure 2As shown in the module structure diagram of the pseudo-query-document scoring model provided in Embodiment 1 of the present invention, the first model input terminal of the pseudo-query-document scoring model is used to receive the pseudo-query Q as input to the model, the second model input terminal is used to receive the document D as input to the model, and the model output terminal is used to output the corresponding score S. The model components of the pseudo-query-document scoring model include: a first preprocessing module, a second preprocessing module, a first encoder, a second encoder, and a scoring module.

[0085] like Figure 2 As shown, the connection relationships of the components of the pseudo-query-document scoring model are as follows: the input end of the first preprocessing module is connected to the input end of the first model, and the output end is connected to the input end of the first encoder; the input end of the second preprocessing module is connected to the input end of the second model, and the output end is connected to the input end of the second encoder; the output end of the first encoder is connected to the first input end of the scoring module; the output end of the second encoder is connected to the second input end of the scoring module; and the output end of the scoring module is connected to the model output end.

[0086] The functions of each component of the pseudo-query-document rating model are shown below.

[0087] 1) First preprocessing module:

[0088] The first preprocessing module of this embodiment of the invention is used to remove stop words, special characters / symbols, and punctuation marks from the pseudo-query Q to obtain the corresponding first preprocessed text; and to perform word segmentation on the first preprocessed text according to the word segmentation rules of the BERT model to obtain the corresponding first word segmentation sequence; and to add a preset classification label CLS to the beginning of the first word segmentation sequence to obtain the corresponding second word segmentation sequence; and to perform embedding encoding according to the embedding encoding rules of the BERT model based on the second word segmentation sequence to obtain the corresponding embedding encoding vector E. Q Send to the first encoder.

[0089] Here, the total number of segments in the first segmentation sequence is denoted as the total number of segments w1, and the total number of segments in the second segmentation sequence is denoted as the total number of segments w2, where w2 = 1 + w1; the embedded encoding vector E Q Composed of w2 sub-encoded vectors e Q,i Composition, 1≤vector index i≤w2.

[0090] 2) Second preprocessing module:

[0091] The second preprocessing module in this embodiment of the invention is used to perform sentence segmentation on document D to obtain multiple corresponding first sentences A. j And regarding the first sentence A j The total number of sentences is counted to obtain the total number of sentences N3, where 1 ≤ sentence index j ≤ N3; and each first sentence A is... jThe second preprocessed text B is obtained by removing stop words, special characters / symbols, and punctuation marks. j And according to the word segmentation rules of the BERT model, each second preprocessed text B... j The corresponding third-order word sequence C is obtained by word segmentation. j ; and in the first third word segmentation sequence C j＝1 Add the classification marker CLS to the head; and add it to the 2nd to N3rd third word segmentation sequence C j Add a preset delimiter marker SEP to the beginning; and add a third segmentation sequence C to N3 completed markers. j The corresponding fourth word segment sequence is obtained by sequentially concatenating the segments; and the corresponding embedding encoding vector E is obtained by embedding the fourth word segment sequence according to the embedding encoding rules of the BERT model. D Send to the second encoder.

[0092] Here, the total number of segments in the fourth segmentation sequence is denoted as the total number of segments w3; the embedded encoding vector E D Composed of w3 sub-encoded vectors e D,k Composition, 1 ≤ vector index k ≤ w3.

[0093] 3) First encoder:

[0094] The first encoder in this embodiment of the invention is used to encode the input embedding vector E. Q High-dimensional semantic feature encoding is performed to obtain the corresponding encoding tensor H. Q Send to the scoring module.

[0095] Here, the first encoder is implemented based on the BERT model. Therefore, the high-dimensional semantic feature encoding process of the first encoder is similar to that of the BERT model, and will not be further elaborated here. It should be noted that the encoding tensor H... Q Composed of w2 sub-encoding vectors h Q,i Composition; where the first sub-encoding vector h Q,i＝1 It corresponds to the classification tag CLS of the second word segmentation sequence.

[0096] 4) Second encoder:

[0097] The second encoder in this embodiment of the invention is used to encode the input embedding vector E. D High-dimensional semantic feature encoding is performed to obtain the corresponding encoding tensor H. D Send to the scoring module.

[0098] Here, the second encoder is implemented based on the BERT model. Therefore, the high-dimensional semantic feature encoding process of the second encoder is similar to that of the BERT model, and will not be further elaborated here. It should be noted that the encoding tensor H...D Composed of w3 sub-encoding vectors h D,k Composition; where the first sub-encoding vector h D,k＝1 This corresponds to the classification label CLS of the fourth word segmentation sequence. Furthermore, the sub-encoding vector h... D,k Feature dimension and sub-encoding vector h Q,i The feature dimensions are consistent.

[0099] 5) Scoring Module:

[0100] The scoring module in this embodiment of the invention is used to encode the tensor H Q The first sub-encoding vector h Q,i＝1 Extract it as the corresponding first vector, and encode the tensor H. D The first sub-encoding vector h D,k＝1 Extract the first vector and use it as the corresponding second vector; calculate the similarity between the first and second vectors based on the cosine similarity algorithm to obtain the corresponding first similarity; and output the first similarity as the corresponding score S.

[0101] Here, since the result of the cosine similarity algorithm is between -1 and 1, the score S is also between -1 and 1.

[0102] Step 2: Randomly select a specified number of N1 target documents from each knowledge domain of the target document library as corresponding N1 seed responses; set a corresponding query text for each seed response, denoted as seed query; and form a corresponding query-feedback seed sample by combining each seed query and the corresponding seed response.

[0103] Wherein, the specified quantity N1 is a positive integer; the total number of query-feedback seed samples is N1×M.

[0104] Here, in step 2, when setting a corresponding query text for each seed feedback, it can be done manually, or it can be obtained by the first main model through reverse question / query generation processing using each seed feedback as the answer / retrieval result.

[0105] Step 3: Configure a large language model instruction template, denoted as the first instruction template, for generating a corresponding feedback when the given query is not empty, and for generating a new query-feedback text pair when the given query is empty, based on a given sample set as a reference; configure a second instruction template for generating a specified number N2 negative feedbacks for a given positive feedback, requiring that the text feature similarity between the positive and negative feedbacks is higher than a preset similarity threshold a but the semantic relevance is lower than a preset relevance threshold b; and configure a third instruction template for extracting query-feedback text pairs using the given retrieval information as the current context.

[0106] Here, the specified quantity N2 is a positive integer; the values ​​of text feature similarity and similarity threshold a range from 0 to 1; the values ​​of semantic relevance and relevance threshold b range from -1 to 1. When semantic relevance is positive, the smaller the value, the weaker the relevance; when it is negative, it indicates that there is a semantic contradiction.

[0107] The configurable parameters of the first instruction template in this embodiment of the invention include a given sample set and a given query; wherein, the given sample set consists of multiple first samples; and the first samples consist of a first sample query and a first sample feedback. The first instruction template is a formatted text, and the specific template format is as follows: Figure 3 The template format diagrams of the first instruction template, the second instruction template, and the third instruction template provided in Embodiment 1 of the present invention are shown, and they consist of multiple text lines, wherein:

[0108] 1) The format of the first line of text is: “Task:” + first task description text. The first task description text is used to prompt the first model to generate corresponding feedback for a given query with reference to the given sample set.

[0109] 2) The line text from the second to the second to last line corresponds one-to-one with the first sample in the given sample set. The line text format of each line is: "sample x:" + "query:" + first sample query + first separator + "feedback:" + first sample feedback + second separator;

[0110] Here, x is the sample number starting from 1 and incrementing by 1 each time; the + in the line text format is the string concatenation operator; the first separator defaults to a semicolon ";", and the second separator defaults to a period ".";

[0111] 3) The text format of the last line is: "Query:" + given query.

[0112] It should be noted that when the given query of the first instruction template in this embodiment of the invention is empty, the first large model will output a corresponding query-feedback text pair in the format of query-feedback text pair of sample x; when the given query of the first instruction template is not empty, the first large model will only output the corresponding feedback text.

[0113] The configurable parameters of the second instruction template in this embodiment of the invention include a given positive feedback. The second instruction template is a formatted text, and the specific template format is as follows: Figure 3 As shown, it consists of 4 lines of text, where:

[0114] 1) The format of the first line of text is: “Task:” + second task description text;

[0115] Here, the second task description text is used to prompt the first model to generate N2 negative feedbacks with similar text features but weak semantic correlation or semantic contradiction for a given positive feedback;

[0116] 2) The format of the second line of text is: "Requirement 1:" + first requirement text;

[0117] Here, the first requirement is that the text feature similarity between each negative feedback generated by the first model and the given positive feedback should be higher than the similarity threshold a, and the value of the text feature similarity is given to be between 0 and 1.

[0118] 3) The text format of the third line is: "Requirement 2:" + the second requirement text;

[0119] Here, the second requirement text requires that the semantic relevance of each negative feedback generated by the first model to a given positive feedback should be lower than the relevance threshold b, and gives the range of semantic relevance values ​​between -1 and 1, and points out that the semantic relevance is positive when there is a weak correlation and negative when there is a semantic contradiction.

[0120] 4) The text format of line 4 is: "Positive Feedback:" + given positive feedback.

[0121] The configurable parameters of the third instruction template in this embodiment of the invention include given search information. The third instruction template is a formatted text, and the specific template format is as follows: Figure 3 As shown, it consists of two text segments, in which:

[0122] 1) The format of the first line of text is: “Task:” + third task description text;

[0123] Here, the third task description text is used to prompt the first major model to extract query-feedback text pairs in the context of the given retrieval information;

[0124] 2) The text format of the second line is: "Search information:" + given search information.

[0125] Step 4: Based on N1×M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model, construct a model training dataset for the pseudo-query-document scoring model, denoted as the first dataset;

[0126] The first dataset includes multiple first data records; each first data record includes a first training query, a first positive feedback, and a first negative feedback sequence; the first negative feedback sequence consists of N2 first negative feedbacks; the first positive feedback and each first negative feedback have similar textual features but are weakly semantically correlated or have semantic contradictions.

[0127] Specifically, this includes: Step 41, setting a first counter initialized to 0;

[0128] Step 42: A corresponding first instruction is obtained by setting the given sample set of the first instruction template as the corresponding N1×M query-feedback seed samples and setting the given query as empty; the first instruction is input into the first large model to generate query-feedback text pairs to obtain the corresponding first query-feedback text pairs; the query text of the first query-feedback text pair is used as a corresponding first training query and the feedback text of the first query-feedback text pair is used as a corresponding first positive feedback.

[0129] Here, the first query-feedback text pair consists of a query text and a feedback text;

[0130] Step 43: Obtain a corresponding second instruction by setting the given positive feedback of the second instruction template as the current first positive feedback; input the second instruction into the first large model for negative feedback generation processing to obtain N2 negative feedbacks that are similar to the current first positive feedback text features but are weakly related or contradictory in meaning; and take each negative feedback obtained this time as a corresponding first negative feedback, and form a corresponding first negative feedback sequence from the N2 first negative feedbacks obtained this time.

[0131] Step 44: A corresponding first data record is formed by the current first training query and its corresponding first positive feedback and first negative feedback sequences;

[0132] Step 45: Increment the first counter by 1; and identify whether the incremented first counter exceeds the preset threshold for the total number of records; if it does not exceed the threshold, return to step 42; if it does exceed the threshold, then the corresponding first dataset is composed of all the obtained first data records.

[0133] Here, the total number of records threshold is a pre-set threshold parameter.

[0134] Step 5: Train the pseudo-query-document rating model based on the first dataset;

[0135] Specifically, it includes: Step 51, dividing the first dataset into two sub-datasets based on a preset first segmentation ratio, denoted as the corresponding first training set and first evaluation set;

[0136] Here, the first segmentation ratio is a pre-set ratio parameter, such as 8:2; both the first training set and the first evaluation set consist of multiple first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0137] Step 52: Calculate a minimum loss l based on the preset temperature parameter τ and the specified quantity N2. min ;

[0138] Here, the temperature parameter τ is a pre-set positive value parameter;

[0139] Minimum loss in the embodiment of the present invention min The calculation formula is:

[0140]

[0141] In the minimum loss l min The + sign appearing in the calculation formula is a plus sign;

[0142] Step 53: Extract the first data record of the first training set as the corresponding current training record;

[0143] Step 54: Input the first training query and the first positive feedback from the current training record as the corresponding pseudo-query Q and document D into the pseudo-query-document scoring model for relevance scoring processing, and use the score S output by the model in this processing as the corresponding positive sample score s. p The first training query and each first negative feedback of the first negative feedback sequence in the current training record are used as the corresponding pseudo-query Q and document D, respectively, and input into the pseudo-query-document scoring model for relevance scoring. The score S output by the model in this processing is used as a corresponding negative sample score s. n,g ;

[0144] Here, 1 ≤ negative sample index g ≤ N2;

[0145] Step 55: Set the maximum positive sample score to 1; and set the obtained positive sample score s... p And N2 negative sample scores s n,g Substitute the preset loss function L; and based on the preset first model optimizer, optimize towards maximizing the positive sample scores s. p The positive sample scores are maximized, and the loss function L is minimized. min In this direction, the model parameters of the first and second encoders of the pseudo-query-document scoring model are fine-tuned.

[0146] Here, the loss function L in this embodiment of the invention is:

[0147]

[0148] In the loss function L, the + sign represents a plus sign.

[0149] The first model optimizer in this embodiment of the invention includes at least the Adam optimizer and the SGD optimizer;

[0150] Step 56: Identify whether the current training record is the last first data record of the first training set; if yes, proceed to step 57; if no, take the next first data record of the first training set as the new current training record and return to step 54.

[0151] Step 57, minimize the loss l min The first label loss is used as the corresponding first label loss; and a round of traversal is performed on all the first data records of the first evaluation set; during this round of traversal, the first data record currently traversed is used as the corresponding current evaluation record; and the first training query and the first positive feedback of the current evaluation record are used as the corresponding pseudo query Q and document D, which are input into the pseudo query-document scoring model for relevance scoring processing, and the score S output by the model in this processing is used as the corresponding positive sample score s. p The first training query and each first negative feedback of the first negative feedback sequence in the current evaluation record are used as the corresponding pseudo-query Q and document D, respectively, and input into the pseudo-query-document scoring model for relevance scoring. The score S output by the model in this processing is used as a corresponding negative sample score s. n,g ; and the positive sample scores s obtained this time p And N2 negative sample scores s n,g The loss function L is substituted into the calculation and the result is used as a corresponding first prediction loss; the current first prediction loss and the first label loss are combined into a corresponding first prediction-label pair; and at the end of this round of traversal, all the obtained first prediction-label pairs are substituted into the preset model evaluation function to calculate the corresponding first evaluation value.

[0152] Here, the model evaluation function in this embodiment of the invention is implemented based on the MAE function, MSE function, or RMSE function;

[0153] Step 58: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 53 to continue training; if it meets the range, stop training and confirm that the model training is complete.

[0154] Here, the first evaluation value range of this embodiment of the invention is a pre-set numerical range.

[0155] Step 6: After model training is completed, the system receives the first query text and the first historical search information input by the user; it then performs query-feedback text pair extraction based on the first historical search information, the third instruction template, and the first large model to obtain the corresponding first text pair sequence; it then performs pseudo-feedback generation based on the first text pair sequence, the first query text, the first instruction template, and the first large model to obtain the corresponding first pseudo-feedback text; the first query text and the first pseudo-feedback text together form the corresponding first pseudo query text; and it then performs dense retrieval based on the first pseudo query text, the pseudo query-document scoring model, and the target document library to obtain the corresponding first search result; and finally, it feeds back the first search result to the current user.

[0156] Specifically, this includes: Step 61, after the model training is completed, receiving the first query text and the first historical search information input by the user;

[0157] Here, the first query text in this embodiment of the invention is the current query text of the current user; the first historical retrieval information consists of all retrieval information of the current user within the most recent specified time period;

[0158] Step 62, and based on the first historical retrieval information, the third instruction template and the first large model, perform query-feedback text pair extraction processing to obtain the corresponding first text pair sequence;

[0159] Specifically, this includes: obtaining a corresponding third instruction by setting the given retrieval information of the third instruction template as the current first historical retrieval information; inputting the third instruction into the first large model to extract query-feedback text pairs to obtain one or more corresponding query-feedback text pairs; forming a corresponding first text pair by combining the query text and feedback text of each extracted query-feedback text pair; and forming a corresponding first text pair sequence by combining all the obtained first text pairs.

[0160] Step 63, and based on the first text sequence, the first query text, the first instruction template and the first large model, perform pseudo feedback generation processing to obtain the corresponding first pseudo feedback text;

[0161] Specifically, this includes: obtaining a corresponding fourth instruction by setting the given sample set of the first instruction template as the current first text pair sequence and setting the given query as the current first query text; inputting the fourth instruction into the first large model for feedback text generation processing and using the feedback text generated by the model in this processing as the corresponding first pseudo-feedback text;

[0162] Step 64, and the first pseudo-query text is composed of the first query text and the first pseudo-feedback text;

[0163] Specifically, this includes: using the first query text as the corresponding first string, using the first pseudo feedback text as the corresponding second string, and using the third string, which is formed by sequentially concatenating the first string, the preset delimiter mark SEP, and the second string, as the corresponding first pseudo query text;

[0164] Here, the first pseudo-query text = the first query text + 'SEP' + the first pseudo-feedback text, where + is the string concatenation operator;

[0165] Step 65, and perform dense retrieval processing based on the first pseudo-query text, the pseudo-query-document scoring model and the target document library to obtain the corresponding first retrieval result;

[0166] Specifically, this includes: taking each target document in the target document library as the corresponding current document, and taking the first pseudo-query text and the current document as the corresponding pseudo-query Q and document D as input to the pseudo-query-document scoring model for relevance scoring processing, and taking the score S output by the model in this processing as a corresponding first document score; taking the largest first document score as the corresponding maximum score; and extracting one or more target documents in the target document library that correspond to the maximum score to form the corresponding first search result;

[0167] Step 66, and then send the first search result back to the current user.

[0168] Figure 4 This is a module structure diagram of a dense retrieval device combining a large language model and pseudo-query technology provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 4 As shown, the device includes: a model preparation module 201, a seed sample preparation module 202, an instruction template preparation module 203, a dataset preparation module 204, a scoring model training module 205, and a dense retrieval module 206.

[0169] The model preparation module 201 is used to select a pre-trained and NLP task-fine-tuned generative large language model as the first large model; and to select a pre-trained BERT series model as the corresponding first encoder and second encoder, and to build a pseudo-query-document scoring model with the first and second encoders as the core; and to use a preset information document library as the corresponding target document library, and to obtain the corresponding total number M by counting the total number of knowledge domains in the target document library; the pseudo-query-document scoring model is used to perform correlation scoring on the pseudo-query Q and document D input to the model and output the corresponding score S, the score S is in the range of -1 to 1; the target document library includes multiple target documents; each target document corresponds to a knowledge domain; the total number M is a positive integer.

[0170] The seed sample preparation module 202 is used to randomly select a specified number N1 target documents from each knowledge domain of the target document library as corresponding N1 seed feedbacks; and set a corresponding query text for each seed feedback as a seed query; and each seed query and the corresponding seed feedback form a corresponding query-feedback seed sample; the specified number N1 is a positive integer; the total number of query-feedback seed samples is N1×M.

[0171] The instruction template preparation module 203 is used to configure a large language model instruction template, denoted as the first instruction template, for generating a corresponding feedback when the given query is not empty, and generating a new query-feedback text pair when the given query is empty, with reference to a given sample set; and to configure a second instruction template for generating a specified number N2 negative feedbacks for a given positive feedback, requiring that the text feature similarity between the positive and negative feedbacks be higher than a preset similarity threshold a but lower than a preset relevance threshold b; and to configure a third instruction template for extracting query-feedback text pairs with the given retrieval information as the current context; the specified number N2 is a positive integer; the values ​​of text feature similarity and similarity threshold a are between 0 and 1; the values ​​of semantic relevance and relevance threshold b are between -1 and 1. When the semantic relevance is positive, the smaller the value, the weaker the relevance; when it is negative, it indicates that there is a semantic contradiction.

[0172] The dataset preparation module 204 is used to construct a model training dataset, denoted as the first dataset, based on N1×M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model, which is a pseudo-query-document scoring model.

[0173] The scoring model training module 205 is used to train the pseudo-query-document scoring model based on the first dataset.

[0174] The dense retrieval module 206 is used to receive the first query text and the first historical retrieval information input by the user after the model training is completed; and to perform query-feedback text pair extraction processing based on the first historical retrieval information, the third instruction template, and the first large model to obtain the corresponding first text pair sequence; and to perform pseudo-feedback generation processing based on the first text pair sequence, the first query text, the first instruction template, and the first large model to obtain the corresponding first pseudo-feedback text; and to form the corresponding first pseudo-query text by combining the first query text and the first pseudo-feedback text; and to perform dense retrieval processing based on the first pseudo-query text, the pseudo-query-document scoring model, and the target document library to obtain the corresponding first retrieval result; and to provide feedback of the first retrieval result to the current user; the first historical retrieval information consists of all retrieval information of the current user in the most recent specified time period.

[0175] The dense retrieval device combining a large language model and pseudo-query technology provided in this embodiment of the invention can execute the method steps in the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0176] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the model preparation module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0177] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0178] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0179] Figure 5 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 5 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0180] exist Figure 5The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0181] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0182] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0183] This invention provides a dense retrieval method, apparatus, electronic device, and computer-readable storage medium that combines a large language model and pseudo-query technology. As described above, this invention pre-selects a pre-trained generative large language model with NLP task fine-tuning as the first large model, and a pre-trained BERT series model as the first and second encoders. A pseudo-query-document scoring model is constructed using the first and second encoders as the core, and a pre-defined information document library is used as the target document library. A specified number of N1 target documents are randomly selected from each knowledge domain of the target document library as seed feedback, and a corresponding query text is set for each seed feedback, denoted as a seed query. Each seed query and its corresponding seed feedback constitute a corresponding query-feedback seed sample. A large language model instruction template is configured to generate a corresponding feedback when the given query is not empty, and a new query-feedback text pair when the given query is empty, denoted as the first instruction template. A specified number of N2 negative feedbacks are configured for a given positive feedback, requiring the text feature similarity between the positive and negative feedbacks to be higher than a preset similarity threshold. A second instruction template with a value of 'a' but semantic relevance lower than a preset relevance threshold 'b' is used. A third instruction template is configured to extract query-feedback text pairs using given retrieval information as the current context. A model training dataset is then constructed for the pseudo-query-document scoring model based on N1×M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model. The pseudo-query-document scoring model is trained based on this dataset. After model training, the first query text and first historical retrieval information input from the user are received. A first text pair sequence is obtained by extracting query-feedback text pairs based on the first historical retrieval information, the third instruction template, and the first large model. A first pseudo-feedback text is generated based on the first text pair sequence, the first query text, the first instruction template, and the first large model. The first pseudo-query text is composed of the first query text and the first pseudo-feedback text. A dense retrieval is performed based on the first pseudo-query text, the pseudo-query-document scoring model, and the target document library to obtain the first retrieval result, which is then fed back to the current user. This embodiment of the invention combines the LLM model, pseudo-query technology, and dense retrieval technology, which not only improves retrieval accuracy but also reduces the probability of retrieval ambiguity.

[0184] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0185] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dense retrieval method combining large language models and pseudo-query techniques, characterized in that, The method includes: A pre-trained generative large language model with NLP task fine-tuning is selected as the first large model; a pre-trained BERT series model is selected as the corresponding first encoder and second encoder, and a pseudo-query-document scoring model is constructed based on the first and second encoders; a preset information document library is used as the corresponding target document library, and the total number of knowledge domains in the target document library is statistically analyzed to obtain the corresponding total number M; the pseudo-query-document scoring model is used to perform correlation scoring on the pseudo-query Q and document D input to the model and output the corresponding score S, where the score S ranges from -1 to 1; the target document library includes multiple target documents; each target document corresponds to a knowledge domain; the total number M is a positive integer; A specified number of N1 target documents are randomly selected from each knowledge domain of the target document library as corresponding N1 seed responses; a corresponding query text is set for each seed response, denoted as a seed query; and each seed query and its corresponding seed response constitute a corresponding query-feedback seed sample; the specified number N1 is a positive integer; the total number of query-feedback seed samples is N1×M; Configure a large language model instruction template, denoted as the first instruction template, for generating a corresponding feedback when a given query is not empty, and generating a new query-feedback text pair when the given query is empty, using a given sample set as a reference; configure a second instruction template for generating a specified number N2 negative feedbacks for a given positive feedback, requiring the text feature similarity between the positive and negative feedbacks to be higher than a preset similarity threshold a but lower than a preset relevance threshold b; configure a third instruction template for extracting query-feedback text pairs using the given retrieval information as the current context; the specified number N2 is a positive integer; the text feature similarity and the similarity threshold a range from 0 to 1; the semantic relevance and the relevance threshold b range from -1 to 1, where a smaller positive value indicates a weaker relevance, and a negative value indicates a semantic contradiction; Based on N1×M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model, a model training dataset is constructed for the pseudo-query-document scoring model, denoted as the first dataset; The pseudo-query-document rating model is trained based on the first dataset; After model training, the system receives the first query text and the first historical retrieval information input by the user; it then performs query-feedback text pair extraction processing based on the first historical retrieval information, the third instruction template, and the first large model to obtain the corresponding first text pair sequence; it then performs pseudo-feedback generation processing based on the first text pair sequence, the first query text, the first instruction template, and the first large model to obtain the corresponding first pseudo-feedback text; the first query text and the first pseudo-feedback text together form the corresponding first pseudo-query text; and it performs dense retrieval processing based on the first pseudo-query text, the pseudo-query-document scoring model, and the target document library to obtain the corresponding first retrieval result; and it then feeds back the first retrieval result to the current user; the first historical retrieval information consists of all retrieval information of the current user within the most recent specified time period.

2. The dense retrieval method combining large language models and pseudo-query techniques according to claim 1, characterized in that, The generative large language model is any one of the following: GPT series models, BERT series models, T5 series models, LLaMA series models, Wenxin series models, or Qwen series models.

3. The dense retrieval method combining large language models and pseudo-query techniques according to claim 1, characterized in that, The first model input terminal of the pseudo-query-document rating model is used to receive the pseudo-query Q input to the model, the second model input terminal is used to receive the document D input to the model, and the model output terminal is used to output the corresponding rating S. The pseudo-query-document scoring model includes a first preprocessing module, a second preprocessing module, a first encoder, a second encoder, and a scoring module; the input of the first preprocessing module is connected to the input of the first model, and the output is connected to the input of the first encoder. The input terminal of the second preprocessing module is connected to the input terminal of the second model, and the output terminal is connected to the input terminal of the second encoder; the output terminal of the first encoder is connected to the first input terminal of the scoring module; the output terminal of the second encoder is connected to the second input terminal of the scoring module; and the output terminal of the scoring module is connected to the output terminal of the model. The first preprocessing module is used to remove stop words, special characters / symbols and punctuation marks from the pseudo query Q to obtain the corresponding first preprocessed text; The first preprocessed text is segmented according to the BERT model's segmentation rules to obtain the corresponding first segmented word sequence; a preset classification label CLS is added to the beginning of the first segmented word sequence to obtain the corresponding second segmented word sequence; and the second segmented word sequence is embedded according to the BERT model's embedding encoding rules to obtain the corresponding embedding encoding vector E. Q Send to the first encoder; the total number of words in the first word segmentation sequence is denoted as the total number of words w1, and the total number of words in the second word segmentation sequence is denoted as the total number of words w2, where w2 = 1 + w1; the embedded encoding vector E Q Composed of w2 sub-encoded vectors e Q,i Composition, 1 ≤ vector index i ≤ w2; The second preprocessing module is used to perform sentence segmentation on the document D to obtain multiple corresponding first sentences A. j And for the first sentence A j The total number of sentences is counted to obtain the corresponding total number of sentences N3, where 1 ≤ sentence index j ≤ N3; and each of the first sentences A j The second preprocessed text B is obtained by removing stop words, special characters / symbols, and punctuation marks. j ; and according to the word segmentation rules of the BERT model, each of the second preprocessed texts B j The corresponding third-order word sequence C is obtained by word segmentation. j ; and in the first of the aforementioned third word segmentation sequences C j＝1 The classification marker CLS is added to the header; And in the 2nd to N3rd third word segmentation sequence C j A preset delimiter marker SEP is added to the beginning of the sequence; and the third word segmentation sequence C with N3 completed markers is added. j The corresponding fourth word segment sequence is obtained by sequentially concatenating the segments; and the corresponding embedding encoding vector E is obtained by embedding the fourth word segment sequence according to the embedding encoding rules of the BERT model. D Send to the second encoder; the total number of segments in the fourth segmentation sequence is denoted as the total number of segments w3; the embedding encoding vector E D Composed of w3 sub-encoded vectors e D,k Composition, 1 ≤ vector index k ≤ w3; The first encoder is used to process the input embedding encoding vector E Q High-dimensional semantic feature encoding is performed to obtain the corresponding encoding tensor H. Q Send to the scoring module; the encoded tensor H Q Composed of w2 sub-encoding vectors h Q,i Composition; the first sub-encoding vector h Q,i＝1 Corresponding to the classification marker CLS of the second word segmentation sequence; The second encoder is used to process the input embedding encoding vector E D High-dimensional semantic feature encoding is performed to obtain the corresponding encoding tensor H. D Send to the scoring module; the encoded tensor H D Composed of w3 sub-encoding vectors h D,k Composition; the sub-encoding vector h D,k The feature dimension and the sub-encoding vector h Q,i The feature dimensions are consistent; the first sub-encoding vector h D,k＝1 It corresponds to the classification tag CLS of the fourth word segmentation sequence; The scoring module is used to process the encoded tensor H. Q The first sub-encoding vector h Q,i＝1 Extract it as the corresponding first vector, and then use the encoded tensor H D The first sub-encoding vector h D,k＝1 The first vector is extracted and used as the corresponding second vector; the similarity between the first and second vectors is calculated based on the cosine similarity algorithm to obtain the corresponding first similarity; and the first similarity is output as the corresponding score S.

4. The dense retrieval method combining large language models and pseudo-query techniques according to claim 1, characterized in that, The configurable parameters of the first instruction template include the given sample set and the given query; the given sample set consists of multiple first samples; the first sample consists of a first sample query and a first sample feedback. The first instruction template is a formatted text consisting of multiple lines. The first line is formatted as "Task:" + first task description text, which prompts the first large model to generate corresponding feedback for the given query based on the given sample set. The second to second-to-last lines correspond one-to-one with the first sample in the given sample set, and each line is formatted as "Sample x:" + "Query:" + first sample query + first separator + "Feedback:" + first sample feedback + second separator, where x is the sample number starting from 1 and incrementing by 1. The "+" in the line format is a character sequence concatenation character. The first separator defaults to a semicolon ";", and the second separator defaults to a period ".". The last line is formatted as "Query:" + given query. When the given query of the first instruction template is empty, the first large model will output a corresponding query-feedback text pair according to the sample format; when the given query of the first instruction template is not empty, the first large model will only output the corresponding feedback text. The configurable parameters of the second instruction template include the given positive feedback; The second instruction template is a formatted text consisting of four lines. The first line is formatted as "Task:" + second task description text, which prompts the first large model to generate N² negative feedbacks with similar text features but weak semantic correlation or semantic contradiction for the given positive feedback. The second line is formatted as "Requirement 1:" + first requirement text, which requires that the similarity between the text features of each negative feedback generated by the first large model and the given positive feedback should be higher than the similarity threshold α. The text feature similarity value is given to be between 0 and 1; the third line of text is formatted as: "Requirement 2:" + second requirement text, which requires that the semantic relevance between each negative feedback generated by the first large model and the given positive feedback should be lower than the relevance threshold b, and gives that the semantic relevance value is between -1 and 1, and indicates that the semantic relevance is positive when there is a weak correlation and negative when there is a semantic contradiction; the fourth line of text is formatted as: "Positive feedback:" + the given positive feedback; The configurable parameters of the third instruction template include the given search information; The third instruction template is a formatted text consisting of two paragraphs; the first line of text is formatted as: "Task:" + third task description text, which is used to prompt the first large model to extract query-feedback text pairs in the context of the given retrieval information; the second line of text is formatted as: "retrieval information:" + given retrieval information; The first dataset includes multiple first data records; the first data records include a first training query, a first positive feedback, and a first negative feedback sequence; the first negative feedback sequence consists of N2 first negative feedbacks; the first positive feedback and each of the first negative feedbacks have similar textual features but are weakly semantically correlated or have semantic contradictions.

5. The dense retrieval method combining large language models and pseudo-query techniques according to claim 4, characterized in that, The step of constructing a model training dataset, denoted as the first dataset, for the pseudo-query-document scoring model based on N1×M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model specifically includes: Step 51: Set a first counter initialized to 0; Step 52: A corresponding first instruction is obtained by setting the given sample set of the first instruction template to the corresponding N1×M query-feedback seed samples and setting the given query to empty; the first instruction is input into the first large model to generate query-feedback text pairs to obtain the corresponding first query-feedback text pairs; the query text of the first query-feedback text pair is used as a corresponding first training query, and the feedback text of the first query-feedback text pair is used as a corresponding first positive feedback. The first query-feedback text pair consists of a query text and a feedback text. Step 53: Obtain a corresponding second instruction by setting the given positive feedback of the second instruction template as the current first positive feedback; input the second instruction into the first large model for negative feedback generation processing to obtain N2 negative feedbacks that are similar to the current first positive feedback text features but are weakly related or semantically contradictory; and take each negative feedback obtained this time as a corresponding first negative feedback, and form the corresponding first negative feedback sequence from the N2 first negative feedbacks obtained this time. Step 54: The current first training query and its corresponding first positive feedback and first negative feedback sequences form a corresponding first data record; Step 55: Increment the first counter by 1; and identify whether the incremented first counter exceeds a preset threshold for the total number of records; if it does not exceed the threshold, return to step 52; if it does exceed the threshold, then the first dataset is composed of all the first data records obtained.

6. The dense retrieval method combining large language model and pseudo-query technology according to claim 4, characterized in that, The step of training the pseudo-query-document rating model based on the first dataset specifically includes: Step 61: Based on a preset first segmentation ratio, the first dataset is divided into two sub-datasets, denoted as the first training set and the first evaluation set. Wherein, both the first training set and the first evaluation set are composed of multiple first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; Step 62: Calculate a minimum loss l based on the preset temperature parameter τ and the specified quantity N2. min ; Wherein, the minimum loss l min The calculation formula is: The minimum loss l min In the calculation formula, + represents a plus sign; Step 63: Extract the first data record of the first training set as the corresponding current training record; Step 64: Input the first training query and the first positive feedback from the current training record as the corresponding pseudo-query Q and document D into the pseudo-query-document scoring model for relevance scoring processing, and use the score S output by the model in this processing as the corresponding positive sample score s. p The first training query and each of the first negative feedbacks in the first negative feedback sequence of the current training record are used as the corresponding pseudo-query Q and document D, respectively, and input into the pseudo-query-document scoring model for correlation scoring processing. The score S output by the model in this processing is used as a corresponding negative sample score s. n,g 1 ≤ negative sample index g ≤ N2; Step 65: Set the maximum value of the positive sample score to 1; and set the obtained positive sample score s... p and N2 negative sample scores s n,g Substitute the preset loss function L; and based on the preset first model optimizer, optimize the model towards improving the positive sample score s. p The positive sample score reaches its maximum value, and the loss function L reaches its minimum loss l. min In the direction of the pseudo-query-document scoring model, the model parameters of the first and second encoders are fine-tuned in one round; Wherein, the loss function L is: In the loss function L, + stands for plus sign; The first model optimizer includes at least the Adam optimizer and the SGD optimizer; Step 66: Identify whether the current training record is the last first data record of the first training set; if yes, proceed to step 67; if no, take the next first data record of the first training set as the new current training record and return to step 64. Step 67, set the minimum loss l min As the corresponding first label loss; and perform a round of traversal on all the first data records of the first evaluation set; and during this round of traversal, take the currently traversed first data record as the corresponding current evaluation record; and take the first training query and the first positive feedback of the current evaluation record as the corresponding pseudo query Q and the document D as input to the pseudo query-document scoring model for relevance scoring processing, and take the score S output by the model in this processing as the corresponding positive sample score s. p The first training query and each of the first negative feedbacks in the first negative feedback sequence of the current evaluation record are used as the corresponding pseudo-query Q and document D, respectively, and input into the pseudo-query-document scoring model for correlation scoring processing. The score S output by the model in this processing is used as a corresponding negative sample score s. n,g ; and the positive sample scores s obtained this time p and N2 negative sample scores s n,g The loss function L is substituted into the calculation and the calculation result is used as a corresponding first prediction loss; the current first prediction loss and the first label loss are combined into a corresponding first prediction-label pair; and at the end of this round of traversal, all the obtained first prediction-label pairs are substituted into the preset model evaluation function to calculate the corresponding first evaluation value. The model evaluation function is implemented based on the MAE function, MSE function, or RMSE function. Step 68: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 63 to continue training; if it meets the range, stop training and confirm that the model training is complete.

7. The dense retrieval method combining large language models and pseudo-query techniques according to claim 4, characterized in that, The step of extracting and processing query-feedback text pairs based on the first historical retrieval information, the third instruction template, and the first large model to obtain the corresponding first text pair sequence specifically includes: A corresponding third instruction is obtained by setting the given retrieval information of the third instruction template to the current first historical retrieval information; the third instruction is input into the first large model to extract query-feedback text pairs to obtain one or more corresponding query-feedback text pairs; the query text and feedback text of each extracted query-feedback text pair are combined to form a corresponding first text pair; and all the obtained first text pairs are combined to form a corresponding first text pair sequence.

8. The dense retrieval method combining large language model and pseudo-query technology according to claim 4, characterized in that, The process of generating the corresponding first pseudo-feedback text based on the first text sequence, the first query text, the first instruction template, and the first large model specifically includes: A corresponding fourth instruction is obtained by setting the given sample set of the first instruction template as the current first text pair sequence and setting the given query as the current first query text; the fourth instruction is then input into the first large model for feedback text generation processing, and the feedback text generated by the model in this processing is used as the corresponding first pseudo feedback text.

9. The dense retrieval method combining large language model and pseudo-query technology according to claim 1, characterized in that, The first pseudo-query text, composed of the first query text and the first pseudo-feedback text, specifically includes: The first query text is used as the corresponding first string, the first pseudo feedback text is used as the corresponding second string, and the third string, which is formed by sequentially concatenating the first string, the preset delimiter mark SEP, and the second string, is used as the corresponding first pseudo query text.

10. The dense retrieval method combining large language model and pseudo-query technology according to claim 1, characterized in that, The process of performing dense retrieval based on the first pseudo-query text, the pseudo-query-document rating model, and the target document library to obtain the corresponding first retrieval result specifically includes: Each target document in the target document library is taken as the corresponding current document, and the first pseudo-query text and the current document are taken as the corresponding pseudo-query Q and document D, respectively. The pseudo-query-document scoring model is then used for relevance scoring, and the score S output by the model in this processing is taken as a corresponding first document score. The largest first document score is taken as the corresponding maximum score. One or more target documents in the target document library that correspond to the maximum score are extracted to form the corresponding first retrieval result.

11. An apparatus for performing the dense retrieval method combining a large language model and pseudo-query technology as described in any one of claims 1-10, characterized in that, The device includes: a model preparation module, a seed sample preparation module, an instruction template preparation module, a dataset preparation module, a scoring model training module, and a dense retrieval module; The model preparation module selects a pre-trained and NLP task-fine-tuned generative large language model as the first large model; selects a pre-trained BERT series model as the corresponding first encoder and second encoder, and constructs a pseudo-query-document scoring model based on the first and second encoders; and uses a preset information document library as the corresponding target document library, and calculates the total number of knowledge domains in the target document library to obtain the corresponding total number M; the pseudo-query-document scoring model performs correlation scoring on the pseudo-query Q and document D input to the model and outputs the corresponding score S, where the score S ranges from -1 to 1; the target document library includes multiple target documents; each target document corresponds to a knowledge domain; and the total number M is a positive integer; The seed sample preparation module is used to randomly select a specified number N1 target documents from each knowledge domain of the target document library as corresponding N1 seed responses; and to set a corresponding query text for each seed response, denoted as a seed query; and to form a corresponding query-feedback seed sample by each seed query and the corresponding seed response; the specified number N1 is a positive integer; the total number of query-feedback seed samples is N1×M; The instruction template preparation module is used to configure a large language model instruction template, denoted as the first instruction template, for generating a corresponding feedback when a given query is not empty, and generating a new query-feedback text pair when the given query is empty, with reference to a given sample set; and to configure a second instruction template for generating a specified number N2 negative feedbacks for a given positive feedback, requiring that the text feature similarity between the positive and negative feedbacks be higher than a preset similarity threshold a but lower than a preset relevance threshold b; and to configure a third instruction template for extracting query-feedback text pairs with the given retrieval information as the current context; the specified number N2 is a positive integer; the text feature similarity and the similarity threshold a range from 0 to 1; the semantic relevance and the relevance threshold b range from -1 to 1, where a smaller positive value indicates a weaker relevance, and a negative value indicates a semantic contradiction; The dataset preparation module is used to construct a model training dataset for the pseudo-query-document scoring model based on N1×M query-feedback seed samples, the first instruction template, the second instruction template, and the first large model, denoted as the first dataset; The scoring model training module is used to train the pseudo-query-document scoring model based on the first dataset; The dense retrieval module is used to receive a first query text and a first historical retrieval information input by the user after model training is completed; and to perform query-feedback text pair extraction processing based on the first historical retrieval information, the third instruction template, and the first large model to obtain a corresponding first text pair sequence; and to perform pseudo-feedback generation processing based on the first text pair sequence, the first query text, the first instruction template, and the first large model to obtain a corresponding first pseudo-feedback text; and to form a corresponding first pseudo-query text by combining the first query text and the first pseudo-feedback text; and to perform dense retrieval processing based on the first pseudo-query text, the pseudo-query-document scoring model, and the target document library to obtain a corresponding first retrieval result; and to feed back the first retrieval result to the current user; the first historical retrieval information consists of all retrieval information of the current user within the most recent specified time period.

12. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-10; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-10.

Images