Data retrieval method and device

Abstract

The invention provides a data retrieval method and device, and relates to the technical field of artificial intelligence. The data retrieval method comprises the following steps: constructing a student model based on a pre-constructed teacher model; performing supervised distillation on the student model by using a pre-constructed offline training data set according to the teacher model until a preset offline training ending condition is met, and obtaining an initial retrieval model; performing cross contrast learning on the teacher model and the initial retrieval model by using a pre-constructed online training data set until a preset online training ending condition is met, and taking the trained initial retrieval model as a target retrieval model; and inputting the to-be-retrieved query data into the target retrieval model to obtain a retrieval result. The student model is constructed according to the teacher model, supervised distillation and cross contrast learning are performed on the student model, compression and acceleration of the model are realized, continuous improvement is performed through instant feedback, and the model is good in performance, high in reasoning speed and low in resource consumption.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a data retrieval method and apparatus. Background Technology

[0002] Current information retrieval methods can be broadly categorized into three types: traditional matching methods, machine learning methods, and large-model methods. Traditional matching methods require a perfect match between the retrieved words or text and the target text to yield accurate results, lacking semantic-level information retrieval capabilities. Machine learning methods can achieve semantic-level information retrieval through training, but obtaining large amounts of supervised data is difficult, and traditional machine learning methods exhibit poor generalization performance. Large-model methods can achieve semantic-level information retrieval based on limited supervised data. However, due to the massive number of model parameters, their application in resource-constrained hardware environments is limited, and their inference computation performance requirements are high, making them unsuitable for resource-constrained environments.

[0003] To reduce the number of model parameters in large models, some studies have employed distillation techniques, enabling student models to achieve faster inference speeds and lower resource consumption while maintaining high performance. However, traditional distillation methods use the scores of the teacher model to guide the student model's learning, and the numerical values ​​of these scores convey very little information, resulting in poor distillation effectiveness. Summary of the Invention

[0004] Based on this, this application provides a data retrieval method and apparatus that enables the retrieval of data with good model performance, fast inference speed, and low resource consumption.

[0005] According to one aspect of this application, a data retrieval method is proposed, comprising: constructing a student model based on a pre-constructed teacher model; using a pre-constructed offline training dataset, performing supervised distillation on the student model according to the teacher model until a preset offline training termination condition is met to obtain an initial retrieval model; using a pre-constructed online training dataset, performing cross-comparison learning on the teacher model and the initial retrieval model until a preset online training termination condition is met, and using the trained initial retrieval model as the target retrieval model; inputting the query data to be retrieved into the target retrieval model to obtain the retrieval results.

[0006] According to some embodiments, constructing a student model based on a pre-built teacher model includes: constructing a base model of the target teacher model based on a pre-built encoder large model; training the base model of the target teacher model using a pre-built first training dataset to obtain the target teacher model; and pruning the target teacher model to obtain the student model.

[0007] According to some embodiments, the encoder big model is obtained by modifying the model structure of a pre-selected decoder big model into bidirectional encoding.

[0008] According to some embodiments, using a pre-built offline training dataset, supervised distillation is performed on the student model based on the teacher model until a preset offline training termination condition is met to obtain an initial retrieval model, including: S21: extracting offline training data from the offline training dataset; S22: inputting the input samples from the offline training data into the teacher model and the student model to obtain a first probability distribution output by the teacher model and a second probability distribution output by the student model; S23: minimizing the difference between the first probability distribution and the second probability distribution using a pre-set supervised learning loss function set, and updating the parameters of the student model; S24: repeating steps S21-S23 until the preset offline training termination condition is met; S25: using the trained student model as the initial retrieval model.

[0009] According to some embodiments, the supervised learning loss function set includes: the InfoNCE loss function, the KL divergence function based on vector scores, and / or the mean square loss function based on vector scores.

[0010] According to some embodiments, step S22 includes: selecting multiple trained encoder models as multiple teacher models; inputting input samples from offline training data into multiple teacher models and student models to obtain multiple initial probability distributions output by multiple teacher models and a second probability distribution output by student models; concatenating the multiple initial probability distributions and using the concatenated result as a first probability distribution.

[0011] According to some embodiments, input samples from offline training data are input into multiple teacher models and student models to obtain multiple initial probability distributions output by multiple teacher models and a second probability distribution output by student models. This includes: based on a nested representation learning mechanism, input samples from offline training data are input into multiple teacher models and student models to obtain multiple first nested vectors output by multiple teacher models and a second nested vector output by student models; from the multiple first nested vectors, nested vectors under a preset dimension are extracted to obtain multiple initial probability distributions corresponding to the preset dimension; from the second nested vectors, nested vectors under a preset dimension are extracted to obtain the second probability distribution corresponding to the preset dimension.

[0012] According to some embodiments, the online training dataset is pre-constructed through the following steps: S31: Inputting documents from the pre-constructed document set and predefined document description prompts into a pre-selected large model to obtain document descriptions, wherein the document description prompts include at least one of the document's applicable role, document's applicable scenario, and document category; S32: Inputting the document descriptions and predefined query generation prompts into the large model to obtain queries corresponding to the documents; S33: Using the queries as input samples and the documents corresponding to the queries as the first output samples to construct initial training data; S34: Based on the pre-constructed vector model, retrieving the input samples in the initial training data to obtain retrieval results, and constructing mining training data based on the retrieval results, wherein the mining training data includes input samples and their corresponding second output samples, and the second output samples include negative examples of the corresponding input samples; S35: Constructing online training data based on the initial training data and the mining training data; S36: Repeating steps S31-S35 to obtain multiple online training data sets, and constructing an online training dataset based on the multiple online training data sets.

[0013] According to some embodiments, step S35 includes: cleaning the initial training data and the mined training data, and updating the initial training data and the mined training data according to the cleaning results, wherein the cleaning includes cleaning based on the reranker model and / or cleaning based on preset rules; extracting the first output sample from the initial training data and using the first output sample as the positive output sample of the input sample; extracting the negative output from the second output sample of the mined training data and using the negative output from the second output sample as the negative output sample of the input sample; and constructing a set of online training data based on the input sample, the positive output sample, and the negative output sample.

[0014] According to some embodiments, using a pre-built online training dataset, a teacher model and an initial retrieval model are cross-comparatively learned until a preset online training termination condition is met. The trained initial retrieval model is then used as the target retrieval model. The process includes: S41: Extracting online training data from the online training dataset; S42: Inputting the input samples of the online training data into the teacher model and the initial retrieval model to obtain a first real-time text vector output by the teacher model and a second real-time text vector output by the initial retrieval model; S43: Minimizing the difference between the first and second real-time text vectors using a pre-set cross-comparative learning loss function set, and updating the parameters of the teacher model and the initial retrieval model. The cross-comparative learning loss function set includes a loss function for aligning the initial retrieval model with the teacher model and a loss function for aligning the teacher model with the initial retrieval model; S44: Repeating steps S41-S43 until the preset online training termination condition is met; S45: Using the trained initial retrieval model as the target retrieval model.

[0015] According to some embodiments, step S42 includes: based on the nested representation learning mechanism, inputting the input samples of online training data into the teacher model and the initial retrieval model to obtain the third nested vector output by the teacher model and the fourth nested vector output by the initial retrieval model; extracting nested vectors under a preset dimension from the third nested vector to obtain the first real-time text vector corresponding to the preset dimension; and extracting nested vectors under a preset dimension from the fourth nested vector to obtain the second real-time text vector corresponding to the preset dimension.

[0016] According to one aspect of this application, a data retrieval apparatus includes: a model building module for building a student model based on a pre-built teacher model; a first training module for supervised distillation of the student model using a pre-built offline training dataset and the teacher model, until a preset offline training termination condition is met, to obtain an initial retrieval model; a second training module for cross-comparison learning of the teacher model and the initial retrieval model using a pre-built online training dataset, until a preset online training termination condition is met, and using the trained initial retrieval model as a target retrieval model; and a model inference module for inputting query data to be retrieved into the target retrieval model to obtain retrieval results.

[0017] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0018] According to one aspect of this application, a computer-readable medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0019] Through the embodiments provided in this application, a student model is constructed based on a teacher model. Offline supervised distillation is then performed on the student model based on the teacher model to obtain an initial retrieval model. This process compresses and accelerates the model, improving retrieval speed and reducing computational complexity and hardware requirements. Subsequently, the teacher model and the initial retrieval model undergo cross-comparison learning, enabling the initial retrieval model to continuously improve through real-time feedback. This achieves bidirectional knowledge transfer and co-evolution between the student and teacher models, allowing the model to quickly adapt to new environments or tasks, and enabling data retrieval with high model performance, fast inference speed, and low resource consumption. Attached Figure Description

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0022] Figure 1 A flowchart of the data retrieval method provided in the embodiments of this application; Figure 2 A flowchart illustrating the construction of a student model based on a pre-built teacher model, provided for embodiments of this application; Figure 3 The flowchart provided in this application describes how a pre-built offline training dataset is used to perform supervised distillation on a student model based on a teacher model until a preset offline training termination condition is met, thus obtaining an initial retrieval model. Figure 4 A flowchart provided for embodiments of this application shows how input samples from offline training data are input into a teacher model and a student model to obtain a first probability distribution output by the teacher model and a second probability distribution output by the student model. Figure 5 A flowchart illustrating how input samples from offline training data are input into multiple teacher models and student models to obtain multiple initial probability distributions output by the multiple teacher models and a second probability distribution output by the student models, provided in an embodiment of this application; Figure 6 A schematic diagram of negative and positive example scores provided for embodiments of this application; Figure 7 The flowchart provided in this application embodiment describes how to use a pre-built online training dataset to perform cross-comparison learning on a teacher model and an initial retrieval model until a preset online training termination condition is met, and how the trained initial retrieval model is used as the target retrieval model. Figure 8 A flowchart illustrating how input samples of online training data are fed into a teacher model and an initial retrieval model to obtain a first real-time text vector output by the teacher model and a second real-time text vector output by the initial retrieval model, provided in an embodiment of this application. Figure 9 A block diagram of a data retrieval device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0027] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0028] For specific implementation details, please refer to the following examples.

[0029] Figure 1 A flowchart illustrating the data retrieval method provided in this application embodiment. For example... Figure 1 As shown, the method includes steps S110-S140.

[0030] In step S110, a student model is constructed based on a pre-built teacher model.

[0031] This application constructs a student model based on the framework of the teacher model. In other words, the student model in this application has the same basic framework as the teacher model, and the student model is derived from the teacher model through modification.

[0032] This application does not restrict the specific method of modification; for example, methods such as pruning or replacing some layers may be used.

[0033] The teacher model is a pre-built model with the required information retrieval capabilities. It can be obtained by directly using publicly available large models or by training on a constructed dataset. This application does not impose any restrictions on this.

[0034] In step S120, using a pre-built offline training dataset, supervised distillation is performed on the student model based on the teacher model until the preset offline training termination condition is met, thus obtaining the initial retrieval model.

[0035] Based on a pre-built offline training dataset, supervised distillation of the student model is performed using distillation techniques to reduce computational resource requirements while maintaining good performance as much as possible. The fine-tuned model is then recorded as the initial retrieval model.

[0036] The preset offline training termination conditions can be set according to the situation, and this application does not impose any restrictions on them, such as the maximum number of iterations, the preset convergence index, etc.

[0037] In step S130, the teacher model and the initial retrieval model are cross-comparatively learned using a pre-built online training dataset until the preset online training termination condition is met, and the trained initial retrieval model is used as the target retrieval model.

[0038] Subsequently, based on the initial retrieval model obtained from offline distillation, further online strategy distillation is performed using a pre-built online training dataset. Specifically, online strategy distillation is a dynamic and real-time knowledge distillation method that utilizes a teacher model and the initial retrieval model for cross-comparative learning. This allows the student model to continuously improve through real-time feedback while learning through practice, making it particularly suitable for scenarios where the model can quickly adapt to new environments or tasks. The distilled model is then designated as the target retrieval model.

[0039] The preset online training termination conditions can be set according to the situation, and this application does not impose any restrictions on them, such as the maximum number of iterations, the preset convergence index, etc.

[0040] In step S140, the query data to be retrieved is input into the target retrieval model to obtain the retrieval results.

[0041] After obtaining the target retrieval model, during the inference phase, the user-inputted query data is fed into the target retrieval model, and the target retrieval model directly outputs the retrieval results.

[0042] This application, through the above-described embodiments, constructs a student model based on a teacher model, performs offline supervised distillation on the student model based on the teacher model to obtain an initial retrieval model, thereby compressing and accelerating the model, improving retrieval speed, reducing computational complexity and hardware requirements; subsequently, cross-compare learning is performed between the teacher model and the initial retrieval model, enabling the initial retrieval model to continuously improve through real-time feedback, achieving bidirectional knowledge transfer and co-evolution between the student model and the teacher model, allowing the model to quickly adapt to new environments or new task scenarios, and achieving data retrieval with good model performance, fast inference speed, and low resource consumption.

[0043] According to some embodiments, refer to Figure 2 In step S110, a student model is constructed based on a pre-built teacher model, which can be specifically implemented through steps S111-S113.

[0044] In step S111, a base model for the target teacher model is constructed based on the pre-built encoder large model.

[0045] It is worth noting that with the rapid development of natural language processing technology, text embedding models play a crucial role in text representation, information retrieval, and text generation tasks. By mapping words, sentences, or documents to a high-dimensional continuous space and using vector representation, text embedding models bring similar texts closer together, which not only improves the manipulability of text data but also significantly enhances the performance of various downstream tasks.

[0046] This application's embodiments use the encoder large model in the text embedding model as the base model for the target teacher model. By using a large language generation model as the model base, the powerful modeling capabilities of the large language model are leveraged to improve retrieval performance.

[0047] It is important to emphasize that the encoder's large model can perform bidirectional encoding, meaning it can simultaneously understand each word based on the context of the entire text, thereby obtaining a deeper and more global semantic representation.

[0048] The encoder large model can be selected from publicly available encoder large models, or it can be constructed from scratch.

[0049] In step S112, the base model of the target teacher model is trained using the pre-constructed first training dataset to obtain the target teacher model.

[0050] The base model is trained based on the pre-built first training dataset to obtain the target teacher model.

[0051] The first training dataset can be a dataset constructed according to the use case, or it can be a public dataset suitable for the use case. This application does not impose any restrictions on this.

[0052] Training methods can be selected according to the circumstances, and this application does not impose any restrictions on them.

[0053] In step S113, the target teacher model is pruned to obtain the student model.

[0054] By using pruning techniques to remove certain layers from the target teacher model, a student model is obtained, thereby reducing complexity and computational cost at the model structure level. Pruning is a model compression technique that reduces the number of neural network parameters and computational cost. By removing unimportant connections or hidden layers, the aim is to reduce model complexity, improve computational efficiency, and, as much as possible, maintain model accuracy.

[0055] The specific layers to be deleted can be selected based on the actual situation.

[0056] According to the example implementation, during pruning, the target teacher model is divided into three parts: front, middle, and back. The middle part is selected through testing, and half of the layers are deleted.

[0057] This application uses the encoder model as the base model for the teacher model, leveraging the powerful capabilities of the large language model to improve retrieval performance. This trains the target teacher model, giving it the robust information retrieval capabilities required for the application scenario. Based on the target teacher model, pruning is performed to obtain the student model. This retains good performance while reducing complexity and computational load at the model structure level, accelerating inference speed and reducing resource consumption.

[0058] According to some embodiments, the encoder big model is obtained by modifying the model structure of a pre-selected decoder big model into bidirectional encoding.

[0059] Specifically, while using the decoder big model as the base model of the teacher model, its one-way generation method is transformed into a two-way encoding method. The resulting model can be understood as the encoder big model, which is used as the base model of the teacher model. The transformed model is more suitable for retrieval scenarios.

[0060] It's worth noting that large language models based on decoder structures excel at generating text content, but when retrieving text content, they generally cannot generate a better vector representation than encoders. Therefore, using a large decoder model as a base model to improve upon an encoder model transforms text content into a vector space, thus enhancing retrieval performance.

[0061] It is understandable that the decoder generates text unidirectionally from left to right, and bidirectional encoding changes the decoding structure to allow it to read text content from right to left.

[0062] According to the example embodiment, the relevant part of the causal mask of the pre-selected decoder large model is removed, and its self-attention mechanism is changed to obtain a model that can perform bidirectional encoding as the encoder large model.

[0063] According to some embodiments, refer to Figure 3 In step S120, using a pre-built offline training dataset, supervised distillation is performed on the student model based on the teacher model until the preset offline training termination condition is met, and the initial retrieval model is obtained. This can be achieved through steps S21-S25.

[0064] In step S21, offline training data is extracted from the offline training dataset.

[0065] Select offline training data from the offline training dataset to begin subsequent training. The offline training data includes input samples and output samples. The output samples include positive and negative examples.

[0066] The offline training dataset can be a publicly available dataset or it can be constructed according to the application scenario; this application does not impose any restrictions on this.

[0067] In step S22, the input samples from the offline training data are input into the teacher model and the student model to obtain the first probability distribution output by the teacher model and the second probability distribution output by the student model.

[0068] Understandably, distillation techniques transfer knowledge from a complex model (often called the teacher model) to a smaller, simpler model (called the student model), aiming to reduce model complexity while maintaining performance. This application employs supervised distillation, combining the output distribution of the teacher model to guide the training of a lighter student model. The aim is to transfer knowledge from the teacher model to the student model, enabling the student model to achieve performance close to or sometimes exceeding that of the teacher model while maintaining a smaller scale.

[0069] In this embodiment of the application, by inputting the input samples of offline training data into the teacher model and the student model, a first probability distribution output by the teacher model and a second probability distribution output by the student model are obtained for subsequent distillation steps.

[0070] Furthermore, in some embodiments, the corresponding output samples of the input samples are used together with the output distribution of the teacher model to guide the training of the student model.

[0071] In step S23, the difference between the first probability distribution and the second probability distribution is minimized using a pre-set set of supervised learning loss functions, and the parameters of the student model are updated.

[0072] It's important to explain that during intensive retrieval, the embedding model transforms the original text into numerical vectors. However, current high-performing embedding models on large-scale text embedding benchmarks (such as the MTEB benchmark) typically have a large number of parameters and high vector dimensions, resulting in slow inference speeds and high storage costs. This poses a challenge to their application in real-world scenarios.

[0073] To address this issue, embodiments of this application propose a multi-stage distillation framework that uses a pre-defined set of supervised learning loss functions to distill knowledge from the teacher model to the student model.

[0074] Specifically, by supervising the learning loss function set, the difference between the first probability distribution and the second probability distribution is minimized, and the parameters of the student model are updated.

[0075] It is important to emphasize that the parameters of the teacher model are frozen at this time.

[0076] Furthermore, in some embodiments, when the student model is trained by using the corresponding output samples of the input samples together with the output distribution of the teacher model, in addition to minimizing the difference between the first probability distribution and the second probability distribution, the supervised learning loss function set is also used to maximize the similarity between the second probability distribution and the positive output samples in the offline training data.

[0077] According to the example embodiment, the supervised learning loss function set includes three carefully designed loss functions that transition from specific constraints to broader constraints. The first is a cosine loss, which calculates the absolute difference between the text representations of the student and teacher models to determine the direction of optimization from the pointwise signals obtained from individual texts. To further avoid overfitting on the training data, a second loss—the similarity loss—is introduced, which measures the semantic differences between the student and teacher models from the perspective of text pairs. Furthermore, a relative similarity distillation loss is designed as a third loss to further utilize relative ranking information. This ensures that the student model learns the ranking preferences of the teacher model on all potential positive and negative text pairs throughout the batch, thereby improving the robustness of embedding learning. It also enables smaller student embedding models to distill multiple larger teacher embedding models.

[0078] In step S24, steps S21-S23 are repeated until the preset offline training termination condition is met.

[0079] Repeat the iteration until training is complete.

[0080] In step S25, the trained student model is used as the initial retrieval model.

[0081] Output the training results, denoted as the initial retrieval model.

[0082] This application employs a supervised distillation method, utilizing a set of supervised learning loss functions to transfer knowledge from the teacher model to the student model, thereby reducing the computational resource requirements of the student model while maintaining good performance as much as possible.

[0083] According to some embodiments, the supervised learning loss function set includes: InfoNCE loss function, KL divergence function based on vector scores, and / or mean square loss function based on vector scores.

[0084] Specifically, the supervised learning loss function set in this application embodiment is used to minimize the difference between the first probability distribution and the second probability distribution, and to maximize the similarity between the second probability distribution and the positive output samples in the offline training data.

[0085] The InfoNCE loss function, short for Information Noise-Contrastive Estimation Loss, is commonly used in self-supervised learning. Its goal is to learn effective representations by maximizing the similarity between positive sample pairs while minimizing the similarity between positive (output) samples and negative (output) samples.

[0086] The expression for the InfoNCE loss function is as follows: ; In the formula, Indicates the first i The representation of the input sample (query text), i.e., the student model's representation of the first input sample (query text). i The second probability distribution of the output of each input sample. Indicates the first i The representation of the main text of the input sample (query text), i.e., the first input sample (query text). i Positive output samples for each input sample (query text). Indicates the first The representation of the negative text of the input sample (query text), that is, the nth input sample (query text). Negative output samples for each input sample (query text). τ It is a temperature parameter used to adjust the smoothness of the distribution. N It refers to the batch size.

[0087] The InfoNCE loss, through contrastive learning, enables the student model to learn representations that can distinguish between positive and negative samples, thereby capturing semantic information from the teacher model's output.

[0088] The KL divergence function based on vector scores is called Kullback-Leibler Divergence. KL divergence, also known as relative entropy, is a measure of the difference between two probability distributions, used to quantify the difference between the student model's output distribution and the teacher model's output distribution.

[0089] The expression for the KL divergence function based on vector fractions is as follows: ; In the formula, p This represents the first probability distribution of the input sample (query text) output in the teacher model; q This represents the second probability distribution of the input sample (query text) output in the student model. Denotes the set of the first probability distribution. Q Describe the set of the second probability distribution. x Indicates the serial number.

[0090] By minimizing the KL divergence between the student model's output distribution Q and the teacher model's output distribution P, the student model can be guided to mimic the teacher model's behavior. In the context of knowledge distillation, the teacher model typically provides a "softer" probability distribution (softmax scaled by temperature), which contains more information than hard labels (i.e., the labels of the input samples), helping the student model learn richer knowledge.

[0091] The Mean Squared Error based on Vector Scores (MSE) is a commonly used regression loss function that measures the average of the squared differences between predicted and true values. In knowledge distillation, it can be used to directly compare the vector representations or scores output by the student and teacher models.

[0092] The expression for the mean squared loss function based on vector scores is as follows: ; In the formula, This represents the output vector or score (i.e., the first probability distribution) of the teacher model. This represents the output vector or score (i.e., the second probability distribution) of the student model; n is the number of samples.

[0093] The former represents the vector result of the query text (from the teacher model), and the latter represents the vector result of the document text (from the student model). By minimizing the MSE, the output score distribution of the student model can be aligned with the output score distribution of the teacher model, thereby optimizing the student model's representation of the text.

[0094] Furthermore, by weighting and summing the multiple loss functions, the total loss function can be obtained.

[0095] Among them, the weights are hyperparameters, which can be selected through experience, or the weights can be adjusted by evaluating the effect after training, based on the selection of experience, so as to select a suitable value.

[0096] In this embodiment, InfoNCE loss emphasizes improving the discriminative power of the student model representation through contrastive learning. KL divergence focuses on matching the probability distribution of the teacher model's output, making it particularly suitable for classification tasks. Mean squared loss directly affects the model's output score or feature vector, aiming to ensure that the student model's output aligns with the teacher model's. By combining these three loss functions, better optimization results are achieved.

[0097] According to some embodiments, refer to Figure 4 In step S22, the input samples from the offline training data are input into the teacher model and the student model to obtain the first probability distribution output by the teacher model and the second probability distribution output by the student model. This can be achieved through steps S221-S223.

[0098] In step S221, multiple trained encoder models are selected as multiple teacher models.

[0099] To further improve the performance of the student model, multiple powerful large-scale embedding models were used as teacher models.

[0100] In this embodiment, multiple trained encoder models are selected as multiple teacher models.

[0101] In step S222, the input samples from the offline training data are input into multiple teacher models and student models to obtain multiple initial probability distributions output by multiple teacher models and a second probability distribution output by student models.

[0102] The input samples from the offline training data are fed into multiple teacher models to obtain multiple initial probability distributions output by the multiple teacher models.

[0103] Meanwhile, the input samples from the offline training data are fed into the student model to obtain the second probability distribution output by the student model.

[0104] In step S223, multiple initial probability distributions are concatenated, and the concatenated result is used as the first probability distribution.

[0105] The vectors (initial probability distributions) generated by all the teacher models are concatenated to create the final ground truth, which serves as the first probability distribution.

[0106] According to the example embodiment, the splicing is performed by connecting the beginning and end.

[0107] This application's embodiments use vectors generated by multiple teacher models to create baseline truth values, thereby creating more accurate learning objects for student models.

[0108] According to some embodiments, refer to Figure 5 In step S420, the input samples from the offline training data are input into multiple teacher models and student models to obtain multiple initial probability distributions output by multiple teacher models and a second probability distribution output by student models. This can be achieved through steps S510-S530.

[0109] In step S510, based on the nested representation learning mechanism, the input samples in the offline training data are input into multiple teacher models and student models to obtain multiple first nested vectors output by multiple teacher models and second nested vectors output by student models.

[0110] Understandably, in order to reduce the vector dimension of the student model, the Matryoshka Representation Learning (MRL) mechanism is used to compress high-dimensional vectors into multiple vectors of different low dimensions.

[0111] The embodiments of this application are particularly applicable to the increase in vector dimension of student models caused by the splicing of teacher model outputs.

[0112] In the specific implementation process, the input samples from the offline training data are fed into multiple teacher models. During processing, each teacher model uses a nested representation learning mechanism to generate nested vector sets, and multiple teacher models output multiple nested vector sets. Each nested vector set is denoted as the first nested vector.

[0113] Similarly, the input samples from the offline training data are input into the student model. During the processing, the student model uses a nested representation learning mechanism to generate a nested vector set, denoted as the second nested vector.

[0114] It needs to be explained that a nested vector set is essentially a series of sub-representations of different dimensions, ranging from coarse to fine, nested within a high-dimensional representation.

[0115] In step S520, nested vectors under a preset dimension are extracted from multiple first nested vectors to obtain multiple initial probability distributions corresponding to the preset dimension.

[0116] Based on the actual situation, set the preset dimensions for extraction, and extract sub-vectors from each first nested vector according to the preset dimensions. The extraction results are denoted as the initial probability distribution. Extraction is performed on multiple first nested vectors to obtain multiple initial probability distributions.

[0117] In step S530, nested vectors under a preset dimension are extracted from the second nested vector to obtain the second probability distribution corresponding to the preset dimension.

[0118] Sub-vectors are extracted from the second nested vector according to a preset dimension, and the result of the extraction is denoted as the second probability distribution.

[0119] The embodiments of this application can flexibly select different dimensions according to the environment configuration. They can use the entire high-dimensional nested vector representation or choose to use only a part of the low-dimensional sub-representation in the nested vector, thus achieving a balance between accuracy and efficiency.

[0120] According to some embodiments, the online training dataset is pre-built through steps S31-S36.

[0121] In step S31, the documents in the pre-built document set and the predefined document description prompts are input into the pre-selected large model to obtain the document description of the document. The document description prompts include at least one of the following: applicable role, applicable scenario, and document category.

[0122] It should be noted that a training data set consists of (query, pos, neg), where pos is a document related to the query (i.e., the model input), and neg is multiple documents unrelated to the query.

[0123] The pre-built document set can be a publicly available document set or a document set built for training tasks; this application does not impose any restrictions on this.

[0124] According to the example implementation, a generic corpus document set is selected as the pre-built document set.

[0125] Document description prompts guide the large model in generating descriptions of documents, including potential stakeholders, scenarios in which stakeholders might be interested in the document, and document categories. Document categories include classifying pos into financial, entity description, medical, policy, scientific, paper abstracts, encyclopedic knowledge, and film categories.

[0126] Document description suggestions include natural language or formatted descriptions that can be used to generate the document descriptions mentioned above. The purpose of document description suggestions is to improve the diversity of generated queries.

[0127] In step S32, the document description and predefined query generation prompts are input into the large model to obtain the query corresponding to the document.

[0128] Query generation suggestions are used to guide the large model to generate diverse queries based on the generated document descriptions.

[0129] In step S33, the query is used as the input sample, and the document corresponding to the query is used as the first output sample to construct the initial training data.

[0130] The generated query is used as the input sample, and the document is used as the output sample (denoted as the first output sample), forming a set of retrieval training data, denoted as the initial training data.

[0131] Initial training data may also include data extracted from publicly available retrieval datasets.

[0132] Furthermore, in some embodiments, step S33 is followed by: All initial training data, including publicly available retrieval datasets and retrieval training data generated by large language models, are cleaned.

[0133] This application does not limit the specific methods of cleaning; its purpose is to remove low-quality and illogical data.

[0134] In step S34, based on the pre-built vector model, the input samples in the initial training data are retrieved to obtain the retrieval results. Based on the retrieval results, mining training data is constructed. The mining training data includes the input samples and their corresponding second output samples. The second output samples include the negative examples of the corresponding input samples.

[0135] Reference Figure 6 When an initial training dataset (query, pos) exists, the difficulty of negative examples will affect the model's learning of retrieval capabilities. Therefore, it is necessary to mine difficult negative examples.

[0136] In the specific implementation process, a suitable vector model with strong retrieval capabilities is used to retrieve the top k relevant documents for each query (i.e., input sample) to obtain the retrieval results. Negative examples are selected from the retrieval results and output, thus constructing retrieval training data with negative examples together with the initial training data, denoted as the mining training data.

[0137] Furthermore, if sufficient computing power is available, some positive examples can be selected for output while negative examples are selected for output, thereby increasing the amount of training data retrieved and enhancing the model's capabilities.

[0138] According to the example implementation, from the first k relevant documents of each query (i.e., input sample), 15 negative documents and 15 positive documents are randomly selected within a certain range to construct the mining training data corresponding to the input sample.

[0139] In step S35, online training data is constructed based on the initial training data and the mined training data.

[0140] The initial training data and the mined training data are combined to form online training data in the format of (query, pos, neg).

[0141] In step S36, steps S31-S35 are repeated to obtain multiple online training data, and an online training dataset is constructed based on the multiple online training data.

[0142] Repeat the steps of constructing online training data to obtain multiple online training data sets, thereby constructing an online training dataset.

[0143] In existing technologies, due to the limited availability of publicly available retrieval data, most high-quality supervised data for specific domains currently requires manual annotation. While this method is accurate, it is costly and time-consuming. This application's embodiments utilize data augmentation techniques to construct a training dataset based on a large language model, thereby enhancing the retrieval capabilities of the vector model.

[0144] It should be emphasized that steps S31-S36 are also applicable to the construction of offline training datasets, and this application will not elaborate on them.

[0145] According to some embodiments, in step S35, online training data is constructed based on the initial training data and the mined training data, which can be specifically implemented through steps S351-S354.

[0146] In step S351, the initial training data and the mined training data are cleaned, and the initial training data and the mined training data are updated according to the cleaning results. The cleaning includes cleaning based on the reranker model and / or cleaning based on preset rules.

[0147] This cleaning step includes cleaning based on the reranker model and cleaning based on preset rules; one or more of these methods can be selected.

[0148] Cleaning based on the Reranker model is particularly effective for positive and / or negative examples obtained from difficult negative example mining. Such data may contain some false positives and false negatives.

[0149] According to the example implementation, among the three reranker models bge-reranker-large (BGE large reranking model), bge-reranker-large-m3 (BGE v2 M3 reranking model), and minicpm-reranker (BGE v2 MiniCPM hierarchical reranking model), bge-reranker-large-m3 is selected as the cleaning model, and the thresholds pos>1 and neg<0 are selected.

[0150] Cleaning based on preset rules is particularly effective for synthetic data, including synthetic queries and their associated positive and negative examples. Such documents may contain illogical content, meaningless characters, and meaningless structured documents (such as tables of contents). To ensure high-quality queries and documents, rule-based cleaning can be performed.

[0151] According to the example implementation, the preset rules include: query length is more than twice that of pos / neg (in general, retrieval is a process of retrieving long documents from short sentences); query content and pos / neg content highly overlap (too simple, affecting the complexity of the data); excessively describing specific details of the document; a single query containing multiple irrelevant issues, etc.

[0152] Based on the cleaned content, update the initial training data and mine the training data.

[0153] In step S352, the first output sample of the initial training data is extracted and used as the positive output sample of the input sample.

[0154] In step S353, negative examples are extracted from the second output sample of the mining training data, and the negative examples in the second output sample are used as negative examples of the input sample.

[0155] In step S354, a set of online training data is constructed based on the input samples, positive output samples, and negative output samples.

[0156] Based on steps S352-S354, integrate the initial training data and the mined training data to construct online input data in (query, pos, neg) format, consisting of a query input (denoted as input sample), a positive output document (denoted as positive output sample), and multiple negative output documents (denoted as negative output samples).

[0157] According to some embodiments, refer to Figure 7In step S130, the teacher model and the initial retrieval model are cross-comparatively learned using a pre-built online training dataset until the preset online training end condition is met. The trained initial retrieval model is then used as the target retrieval model. This can be achieved through steps S41-S45.

[0158] In step S41, online training data is extracted from the online training dataset.

[0159] Select online training data from the online training dataset to begin subsequent training. The online training data includes input samples and output samples. The output samples include positive output samples and negative output samples.

[0160] In step S42, the input samples of the online training data are input into the teacher model and the initial retrieval model to obtain the first real-time text vector output by the teacher model and the second real-time text vector output by the initial retrieval model.

[0161] Understandably, the initial retrieval model was treated as a student model during this online distillation process.

[0162] In this embodiment of the application, by inputting the input samples of online training data into the teacher model and the initial retrieval model, a first real-time text vector output by the teacher model and a second real-time text vector output by the initial retrieval model are obtained for subsequent distillation steps.

[0163] It should be noted that, in order to save training time, the first real-time text vector and the second real-time text vector are intermediate output representations of the model.

[0164] In step S43, the difference between the first real-time text vector and the second real-time text vector is minimized using a pre-set set of cross-contrast learning loss functions, and the parameters of the teacher model and the initial retrieval model are updated. The set of cross-contrast learning loss functions includes a loss function for aligning the initial retrieval model with the teacher model and a loss function for aligning the teacher model with the initial retrieval model.

[0165] Specifically, by cross-comparing and learning a set of loss functions, the difference between the first real-time text vector and the second real-time text vector is minimized, and the parameters of the teacher model and the initial retrieval model are updated. By allowing the small model to learn and practice simultaneously, it is continuously improved through real-time feedback.

[0166] The cross-comparison learning loss function set includes the loss function for aligning the initial retrieval model with the teacher model and the loss function for aligning the teacher model with the initial retrieval model.

[0167] Understandably, in this online distillation process, the initial retrieval model is treated as a student model. The core idea is to achieve bidirectional knowledge transfer and co-evolution between the student and teacher models through a loss function learned by cross-comparison. This bidirectional learning allows both models to benefit from each other's outputs, learning through the process of the student model aligning with the teacher model and the teacher model aligning with the student model.

[0168] This application does not impose specific restrictions on the set of cross-contrast learning loss functions; they can be either the InfoNCE loss function or the KL divergence.

[0169] According to the example embodiment, the cross-contrast learning loss function set includes the InfoNCE loss function for cross-contrast learning, specifically including: The loss function for aligning the initial retrieval model with the teacher model: ; In the formula, S represents the initial retrieval model. Indicates the initial retrieval model S for the first... k The real-time text vector output of one input sample, where T represents the teacher model. Indicates teacher model T for the first k Real-time text vector output for each input sample. Indicates teacher model T for the first b Real-time text vector output of each input sample; τ is a temperature parameter used to adjust the smoothness of the similarity calculation; | B | represents the batch size.

[0170] The goal of the loss function aligning the initial retrieval model with the teacher model is to ensure that the output of the initial retrieval model S better matches the output of the teacher model T. This maximizes the similarity between the representation of the initial retrieval model S and the corresponding representation of the teacher model T, while minimizing the similarity to the representations of other teacher models in the current batch. This means that the initial retrieval model is learning how to "imitate" the output of the teacher models, thereby absorbing the knowledge from the teacher models.

[0171] The loss function of the teacher model is aligned with the initial retrieval model: ; In the formula, S represents the initial retrieval model. Indicates the initial retrieval model S for the first... k The real-time text vector output of one input sample, where T represents the teacher model. Indicates teacher model T for the first k Real-time text vector output for each sample; Indicates teacher model T for the first bReal-time text vector output of each input sample; τ is a temperature parameter used to adjust the smoothness of the similarity calculation; | B | represents the batch size.

[0172] The goal of the loss function that aligns the teacher model with the initial retrieval model is to ensure that the output of the teacher model T aligns with the output of the initial retrieval model S. This maximizes the similarity between the representation of the teacher model T and the corresponding representation of the student model S, while minimizing the similarity with the representations of other student models in the current batch. This indicates that the teacher model is also learning from the student models; for example, the student models may explore new and more effective strategies in certain aspects, thereby helping the teacher model to improve or adjust its own strategies.

[0173] In step S44, steps S41-S43 are repeated until the preset online training end condition is met.

[0174] Repeat the iteration until training is complete.

[0175] In step S45, the trained initial retrieval model is used as the target retrieval model.

[0176] The output of the training result is denoted as the target retrieval model.

[0177] This application uses the initial retrieval model as a student model, directly learning the text representations generated by the teacher model. During model training, the loss is calculated through cross-alignment using the intermediate outputs of the teacher model and the initial retrieval model, thus performing online policy distillation. The model can be adjusted in real-time based on the latest data and performance, improving its adaptability and generalization ability. The initial retrieval model not only learns from the teacher model, but the teacher model also acquires information from the initial retrieval model, forming a co-evolutionary cycle. This is highly beneficial for exploring new solutions or maintaining optimal model performance in dynamic environments. In some continuous learning scenarios, online distillation can help the model learn new tasks while retaining knowledge from old tasks.

[0178] According to some embodiments, refer to Figure 8 In step S42, the input samples of the online training data are input into the teacher model and the initial retrieval model to obtain the first real-time text vector output by the teacher model and the second real-time text vector output by the initial retrieval model. This can be achieved through steps S810-S830.

[0179] In step S810, based on the nested representation learning mechanism, the input samples of the online training data are input into the teacher model and the initial retrieval model to obtain the third nested vector output by the teacher model and the fourth nested vector output by the initial retrieval model.

[0180] Understandably, in order to reduce the vector dimension of the student model, the Matryoshka Representation Learning (MRL) mechanism is used to compress high-dimensional vectors into multiple vectors of different low dimensions.

[0181] In the specific implementation process, the input samples from the online training data are input into the teacher model. During the processing, the teacher model adopts a nested representation learning mechanism to generate a nested vector set, denoted as the third nested vector.

[0182] Similarly, the input samples from the online training data are fed into the initial retrieval model. During the processing, the initial retrieval model uses a nested representation learning mechanism to generate a nested vector set, denoted as the fourth nested vector.

[0183] It needs to be explained that a nested vector set is essentially a series of sub-representations of different dimensions, ranging from coarse to fine, nested within a high-dimensional representation.

[0184] In step S820, nested vectors under a preset dimension are extracted from the third nested vector to obtain the first real-time text vector corresponding to the preset dimension.

[0185] Set the preset dimensions for extraction based on the actual situation, and extract sub-vectors from the third nested vector according to the preset dimensions. The extracted result is recorded as the first real-time text vector.

[0186] In step S830, nested vectors under a preset dimension are extracted from the fourth nested vector to obtain the second real-time text vector corresponding to the preset dimension.

[0187] Sub-vectors are extracted from the fourth nested vector according to the preset dimensions, and the extracted result is denoted as the second real-time text vector.

[0188] The embodiments of this application can flexibly select different dimensions according to the environment configuration. They can use the entire high-dimensional nested vector representation or choose to use only a part of the low-dimensional sub-representation in the nested vector, thereby speeding up the training and improving the model performance.

[0189] According to the example embodiment, the performance of the model before and after distillation using the model distillation method of steps S110-S130 provided in this application was evaluated, and the evaluation results are shown in Table 1.

[0190] Table 1 Evaluation results of distillation

[0191] As can be seen, compared with the undistilled model, the model obtained by distillation using the model distillation method of steps S110-S130 provided in this application is improved, demonstrating the effectiveness of the method provided in this application.

[0192] This application overcomes the shortcomings of existing technologies in text retrieval, such as slow model performance, high vector dimension, and mediocre results. It not only has significant advantages in terms of miniaturization and efficiency of large language retrieval models, but also provides a solid foundation for future wider retrieval applications and practical implementation.

[0193] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0194] Figure 9 A block diagram of a data retrieval apparatus according to an exemplary embodiment is shown.

[0195] Figure 9 The apparatus shown can perform the data retrieval method described above according to the embodiments of this application.

[0196] like Figure 9 As shown, the data retrieval device may include: See Figure 9 Referring to the preceding description, the model building module 910 is used to build student models based on pre-built teacher models.

[0197] The first training module 920 is used to perform supervised distillation on the student model based on the teacher model using a pre-built offline training dataset until the preset offline training termination condition is met, thus obtaining the initial retrieval model.

[0198] The second training module 930 is used to cross-compare and learn the teacher model and the initial retrieval model using a pre-built online training dataset until the preset online training termination condition is met, and then the trained initial retrieval model is used as the target retrieval model.

[0199] The model reasoning module 940 is used to input the query data to be retrieved into the target retrieval model and obtain the retrieval results.

[0200] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0201] This application discloses an electronic device, including: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to execute the above-described instruction generation method.

[0202] For example, refer to Figure 10 , Figure 10The illustrated electronic device 1000 includes a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, for example, via a bus 1002. Optionally, the electronic device 1000 may further include a transceiver 1004. It should be noted that in practical applications, the transceiver 1004 is not limited to one type, and the structure of this electronic device 1000 does not constitute a limitation on the embodiments of the present invention.

[0203] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in this disclosure. Processor 1001 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0204] Bus 1002 may include a pathway for transmitting information between the aforementioned components. Bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0205] The memory 1003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other storage medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0206] The memory 1003 is used to store application code that executes the present invention, and its execution is controlled by the processor 1001. The processor 1001 is used to execute the application code stored in the memory 1003 to implement the content shown in the foregoing method embodiments.

[0207] Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0208] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to execute an instruction generation method.

[0209] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0210] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data retrieval method, characterized in that, include: Construct student models based on pre-built teacher models; Using a pre-built offline training dataset, supervised distillation is performed on the student model based on the teacher model until the preset offline training termination condition is met, thus obtaining the initial retrieval model. Using a pre-built online training dataset, the teacher model and the initial retrieval model are cross-comparatively learned until the preset online training termination condition is met, and the trained initial retrieval model is used as the target retrieval model. Input the query data to be retrieved into the target retrieval model to obtain the retrieval results.

2. The method according to claim 1, characterized in that, The student model is constructed based on a pre-built teacher model, including: Based on the pre-built encoder model, a base model for the target teacher model is constructed. Using a pre-constructed first training dataset, the base model of the target teacher model is trained to obtain the target teacher model; The target teacher model is pruned to obtain the student model.

3. The method according to claim 2, characterized in that, The encoder big model is obtained by modifying the model structure of a pre-selected decoder big model into a bidirectional encoding.

4. The method according to claim 1, characterized in that, Using a pre-built offline training dataset, supervised distillation is performed on the student model based on the teacher model until a preset offline training termination condition is met, resulting in an initial retrieval model, including: S21: Extract offline training data from the offline training dataset; S22: Input the input samples from the offline training data into the teacher model and the student model to obtain the first probability distribution output by the teacher model and the second probability distribution output by the student model; S23: Using a pre-set set of supervised learning loss functions, minimize the difference between the first probability distribution and the second probability distribution, and update the parameters of the student model; S24: Repeat steps S21-S23 until the preset offline training end condition is met; S25: Use the trained student model as the initial retrieval model.

5. The method according to claim 4, characterized in that, The supervised learning loss function set includes: InfoNCE loss function, KL divergence function based on vector scores, and / or mean square loss function based on vector scores.

6. The method according to claim 4, characterized in that, Step S22 includes: Select multiple large encoder models that have been trained, and use them as multiple teacher models; Input samples from the offline training data are input into the multiple teacher models and the student model to obtain multiple initial probability distributions output by the multiple teacher models and a second probability distribution output by the student model; The multiple initial probability distributions are concatenated, and the concatenated result is used as the first probability distribution.

7. The method according to claim 6, characterized in that, Input samples from the offline training data are fed into the multiple teacher models and the student model to obtain multiple initial probability distributions output by the multiple teacher models and a second probability distribution output by the student model, including: Based on the nested representation learning mechanism, the input samples in the offline training data are input into the multiple teacher models and the student model to obtain multiple first nested vectors output by the multiple teacher models and second nested vectors output by the student model; From the plurality of first nested vectors, extract the nested vectors under the preset dimension to obtain the plurality of initial probability distributions corresponding to the preset dimension; From the second nested vector, extract the nested vector under the preset dimension to obtain the second probability distribution corresponding to the preset dimension.

8. The method according to claim 1, characterized in that, The online training dataset is pre-built through the following steps: S31: Input the documents in the pre-built document set and the predefined document description prompts into the pre-selected large model to obtain the document description of the document, wherein the document description prompts include at least one of the following: applicable role of the document, applicable scenario of the document, and document category; S32: Input the document description and predefined query generation prompts into the large model to obtain the query corresponding to the document; S33: Using the query as the input sample and the document corresponding to the query as the first output sample, construct the initial training data; S34: Based on the pre-built vector model, the input samples in the initial training data are retrieved to obtain retrieval results, and mining training data is constructed according to the retrieval results, wherein the mining training data includes the input samples and their corresponding second output samples, and the second output samples include negative examples of the corresponding input samples. S35: Construct online training data based on the initial training data and the mined training data; S36: Repeat steps S31-S35 to obtain multiple online training data, and construct the online training dataset based on the multiple online training data.

9. The method according to claim 8, characterized in that, Step S35 includes: The initial training data and the mined training data are cleaned, and the initial training data and the mined training data are updated according to the cleaning results. The cleaning includes cleaning based on the reranker model and / or cleaning based on preset rules. Extract the first output sample from the initial training data, and use the first output sample as the positive output sample of the input sample; Extract the negative examples from the second output sample of the mining training data, and use the negative examples from the second output sample as the negative examples of the input sample; Based on the input samples, the positive output samples, and the negative output samples, a set of online training data is constructed.

10. The method according to claim 1 or 9, characterized in that, Using a pre-built online training dataset, the teacher model and the initial retrieval model are cross-trained and compared until a preset online training termination condition is met. The trained initial retrieval model is then used as the target retrieval model, including: S41: Extract online training data from the online training dataset; S42: Input the input samples of the online training data into the teacher model and the initial retrieval model to obtain the first real-time text vector output by the teacher model and the second real-time text vector output by the initial retrieval model; S43: Using a pre-set set of cross-contrast learning loss functions, minimize the difference between the first real-time text vector and the second real-time text vector, and update the parameters of the teacher model and the initial retrieval model, wherein the set of cross-contrast learning loss functions includes a loss function for aligning the initial retrieval model with the teacher model and a loss function for aligning the teacher model with the initial retrieval model; S44: Repeat steps S41-S43 until the preset online training end condition is met; S45: Use the trained initial retrieval model as the target retrieval model.

11. The method according to claim 10, characterized in that, Step S42 includes: Based on the nested representation learning mechanism, the input samples of the online training data are input into the teacher model and the initial retrieval model to obtain the third nested vector output by the teacher model and the fourth nested vector output by the initial retrieval model; From the third nested vector, extract the nested vector under the preset dimension to obtain the first real-time text vector corresponding to the preset dimension; From the fourth nested vector, the nested vector under the preset dimension is extracted to obtain the second real-time text vector corresponding to the preset dimension.

12. A data retrieval device, characterized in that, include: The model building module is used to build student models based on pre-built teacher models; The first training module is used to perform supervised distillation on the student model based on the teacher model using a pre-built offline training dataset until the preset offline training termination condition is met, so as to obtain the initial retrieval model. The second training module is used to perform cross-comparison learning on the teacher model and the initial retrieval model using a pre-built online training dataset until the preset online training end condition is met, and the trained initial retrieval model is used as the target retrieval model. The model reasoning module is used to input the query data to be retrieved into the target retrieval model to obtain the retrieval results.

13. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-11.

14. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-11.

Images