Methods, apparatus, equipment, media, and programs for generating training data

Abstract

This application provides a method, apparatus, device, medium, and program product for generating training data, which may involve artificial intelligence technology and computer technology. The method includes: inputting a first document from a data source corresponding to a vertical domain text retrieval model into a first large language model to obtain a query statement corresponding to the first document; querying the query statement in the data source to retrieve N second documents; where N is an integer greater than 1; selecting M target second documents from the N second documents that constitute negative samples with the query statement; where M is a positive integer; and for each of the M target second documents, constructing a triplet training data with a partial order relation for a general text retrieval model using the query statement, the first document, and the target second document. This automatic generation method of training data can improve the efficiency and accuracy of training data generation compared to manual methods.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method, apparatus, device, medium and program product for generating training data. Background Technology

[0002] Text retrieval models play an important role in information retrieval and natural language processing, primarily used to quickly and accurately find documents relevant to a query from a large number of documents.

[0003] In the text retrieval process, the text retrieval model can be understood as a logically significant Siamese network, consisting of two identical networks. Each of these networks can be a transformer-based encoder, such as the Bidirectional Encoder Representations from Transformers (BERT) model. Therefore, current text retrieval methods can be understood as matching methods based on encoder-based embedding representations. Specifically, two encoders can be used to embed the query and document into their representations. The similarity between the two embeddings is then calculated, and this similarity is used as the semantic matching score between the query and the document. Finally, documents with higher semantic matching scores can be retrieved.

[0004] For text retrieval models, a large amount of training data is inevitably needed to iteratively optimize the model. Currently, training data is mainly constructed manually. However, for text retrieval models in vertical fields, the construction of training data requires higher levels of expertise from personnel. This manual approach inevitably leads to problems such as low efficiency and low accuracy in training data generation. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program product for generating training data, thereby improving the efficiency and accuracy of training data generation.

[0006] In a first aspect, embodiments of this application provide a method for generating training data, comprising: inputting a first document from a data source corresponding to a vertical domain text retrieval model into a first large language model to obtain a query statement corresponding to the first document; querying the query statement in the data source to retrieve N second documents; wherein N is an integer greater than 1; selecting M target second documents from the N second documents that constitute negative samples with the query statement; wherein M is a positive integer; and for each of the M target second documents, constructing a triplet training data with a partial order relation for a general text retrieval model by combining the query statement, the first document, and the target second document.

[0007] Secondly, embodiments of this application provide a training data generation apparatus, comprising: an input module, a recall module, a selection module, and a generation module. The input module is used to input a first document from a data source corresponding to a vertical domain text recall model into a first large language model to obtain a query statement corresponding to the first document. The recall module is used to query the query statement in the data source to recall N second documents, where N is an integer greater than 1. The selection module is used to select M target second documents from the N second documents that constitute negative samples with the query statement, where M is a positive integer. The generation module is used to, for each of the M target second documents, construct a triplet training data with a partial order relationship for a general text recall model, consisting of the query statement, the first document, and the target second document.

[0008] In some implementations, the training data generation device further includes: a sorting module, used to sort the M target second documents based on the similarity between the query statement and the M target second documents; the generation module is also used to: for any two target second documents among the M target second documents, construct a triplet training data with a partial order relationship for a general text recall model using the query statement and the arbitrary two target second documents, and generate labels for the triplet training data corresponding to the arbitrary two target second documents based on the order of the M target second documents; wherein, the labels are used to identify whether the triplet training data corresponding to the arbitrary two target second documents is a positive sample pair or a negative sample pair.

[0009] In some implementations, the generation module is specifically used to: generate labels to identify that the training data of the triplets corresponding to any two target second documents are negative sample pairs if the difference in ranking between any two target second documents is greater than a preset threshold; or, generate labels to identify that the training data of the triplets corresponding to any two target second documents are positive sample pairs if the difference in ranking between any two target second documents is less than or equal to the preset threshold.

[0010] In some implementations, the sorting module is specifically used to: input M target second documents and query statements into a second large language model, and sort the M target second documents based on the similarity between the query statement and the M target second documents.

[0011] In some implementations, the recall module is specifically used to: query the query statement in the data source, and recall the first N second documents in descending order of similarity between the query statement and the documents in the data source.

[0012] In some implementations, the recall module is specifically used to: input the query statement and the documents in the data source into the third language model, and recall the top N second documents in descending order of similarity between the query statement and the documents in the data source.

[0013] In some implementations, the selection module is specifically used to: select M second documents from the Pth to the Nth second documents in descending order of similarity between the query statement and the N second documents, as the M target second documents; or, select the last M second documents from the N second documents in descending order of similarity between the query statement and the N second documents, as the M target second documents.

[0014] In some implementations, the selection module is specifically used to: randomly select M second documents from the Pth to the Nth second documents.

[0015] In some implementations, the selection module is specifically used to select the last M second documents from the Pth to the Nth second documents.

[0016] In some implementations, the training data generation apparatus further includes a training module for training the general text recall model based on training data of triples with partial order relations of the general text recall model, to obtain a vertical domain text recall model.

[0017] In some implementations, the training data generation device further includes a preprocessing module, used to preprocess the first document before the input module inputs the first document from the data source corresponding to the vertical domain text retrieval model into the first large language model to obtain the query statement corresponding to the first document; correspondingly, the preprocessing module is specifically used to input the preprocessed first document into the first large language model to obtain the query statement.

[0018] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the methods as described in the first aspect or its various implementations.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0020] Fifthly, embodiments of this application provide a computer program product including computer program instructions that cause a computer to perform the methods as described in the first aspect or its various implementations.

[0021] Sixthly, embodiments of this application provide a computer program that causes a computer to perform the methods as described in the first aspect or its various implementations.

[0022] The technical solution provided in this application involves inputting a first document from the data source corresponding to the vertical domain text retrieval model into a first large language model to obtain the query statement corresponding to the first document; querying the query statement in the data source to retrieve N second documents, where N is an integer greater than 1; selecting M target second documents from the N second documents that form negative samples with the query statement, where M is a positive integer; and for each of the M target second documents, constructing a triplet training data with a partial order relation for the general text retrieval model using the query statement, the first document, and the target second document. Since the query statement, the first document, and the target second documents are all automatically obtained, automated generation of training data is achieved. This automatic generation method for training data improves the efficiency and accuracy of training data generation compared to manual methods. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0025] Figure 2A and Figure 2B This is a schematic diagram of the interface of the intelligent customer service product provided in the embodiments of this application;

[0026] Figure 3 A flowchart illustrating a method for generating training data provided in an embodiment of this application;

[0027] Figure 4 A schematic diagram illustrating a small-model distillation training process provided in an embodiment of this application;

[0028] Figure 5A schematic diagram illustrating a training data generation process provided in an embodiment of this application;

[0029] Figure 6 A schematic diagram illustrating a model training process provided in an embodiment of this application;

[0030] Figure 7 A schematic diagram of a training data generation apparatus 700 provided in an embodiment of this application;

[0031] Figure 8 This is a schematic block diagram of the electronic device 800 provided in the embodiments of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0034] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0035] Before introducing the technical solution of this application, the relevant knowledge of this application will be explained below:

[0036] I. Text Retrieval Models play a crucial role in information retrieval and natural language processing, primarily used to quickly and accurately find documents relevant to a query from a large volume of documents. Text retrieval models are widely applied in search engines, recommendation systems, question-answering systems, and other fields.

[0037] II. Vertical domain text retrieval model: This model uses machine learning or deep learning techniques to process and analyze text data in a specific domain in order to achieve rapid retrieval of relevant text.

[0038] Third, the general text retrieval model is a model that can quickly find text related to a user's query or needs within a broad collection of knowledge or text. This model is not limited to a specific field or industry, but has a wider range of applications.

[0039] Fourth, small-model distillation training involves training a small, simple model (student model) to replicate and learn the knowledge and performance of a large model (teacher model). This technique achieves knowledge transfer by having the student model mimic the output of the teacher model, enabling the student model to perform reasoning on a smaller scale while maintaining performance close to or even better than the teacher model, thus improving reasoning efficiency.

[0040] 5. Text matching, used to determine whether two texts express the same semantics.

[0041] VI. Large Language Models (LLMs) are a type of artificial intelligence model designed to understand and generate human language. They primarily refer to language models trained on massive text corpora, containing billions of parameters, and are deep learning models used for natural language processing tasks. Through extensive training on large datasets and with a vast number of parameters, these models can capture most of the syntax and semantics of human language, thus performing exceptionally well in various natural language processing tasks.

[0042] VII. Embedding representation, also known as word embedding or embedding vector, is essentially a method for converting complex data objects (such as words, phrases, images, users, etc.) into low-dimensional, continuous vector representations. These vectors are typically used to capture semantic relationships or similarities between data objects. In machine learning and deep learning models, embedding layers learn the latent features of the data, mapping high-dimensional, discrete data to a low-dimensional, continuous vector space, thereby improving the model's processing efficiency and accuracy.

[0043] 8. Siamese Network is a neural network architecture used to solve similarity-based comparison tasks. Its basic idea is to simultaneously feed input data into two identical neural networks, which share the same weights and parameters. Through these two networks, embedding representations of the two input data can be obtained separately. Furthermore, the similarity between the two embedding representations can be calculated.

[0044] 9. An encoder is used to transform an input sequence (usually text) into a set of feature representations (also called encodings). These feature representations contain semantic information about the input sequence, which the decoder references when generating the output sequence.

[0045] 10. The BERT model is a pre-trained language representation model that uses a bidirectional encoder to simultaneously consider the left and right contextual information of each word in the text, thereby more accurately understanding the meaning of the text.

[0046] 11. Partial Order Relation: Let R be a binary relation on set A. If R satisfies the following three properties, then R is called a partial order relation on A, usually denoted by "≤":

[0047] Reflexivity: For any x∈A, xRx. That is, all vertices in the relation graph have cycles.

[0048] Antisymmetry: For any x, y∈A, if x∈Ay and y∈Ax, then x=y. That is, there are 0 or 1 directed edges between two vertices.

[0049] Transitivity: For any x, y, z ∈ A, if x ∈ y and y ∈ z, then x ∈ z. That is, if the premises a→b and b→c hold, then a→c must exist.

[0050] The partial order relation can be represented by the symbol "≤", which is read as "less than or equal to", but this does not necessarily mean "less than or equal to" in the general sense. If x≤y, it can also be said that x comes before y.

[0051] A partial order relation, also known as a partial order relation, is a binary relation in mathematics. It describes a "not greater than" or "not less than" relationship between a set of elements, but this relationship does not necessarily mean that all elements are comparable. Compared to a total order relation (or linear order relation), a partial order relation allows for the existence of incomparable pairs of elements.

[0052] 12. Partially ordered set: A set that has a partially ordered relation is called a partially ordered set (poset).

[0053] Thirteen, positive samples refer to samples whose predicted values ​​match the true labels.

[0054] Fourteen, negative samples refer to samples whose predicted values ​​do not match the true labels.

[0055] 15. Positive sample pairs refer to similar sample pairs.

[0056] It should be understood that positive sample pairs refer to sample pairs that should be close to each other in the feature space, i.e., similar or identical samples. These sample pairs usually come from the same category or have some kind of similarity. During training, the model strives to reduce the distance between positive sample pairs to enhance their similarity. The definition of a positive sample pair can vary depending on the task and dataset. For example, in a face recognition task, a positive sample pair might be different facial images of the same person; in an image retrieval task, a positive sample pair might be images with similar or related content. In other words, a positive sample pair refers to a sample pair with a similarity greater than a preset similarity.

[0057] Sixteen, negative sample pairs refer to dissimilar sample pairs.

[0058] It should be understood that negative sample pairs refer to sample pairs that should be far apart from each other in the feature space, i.e., dissimilar or different samples. These sample pairs usually come from different categories or have significant differences. During training, the model strives to increase the distance between negative sample pairs to reduce their similarity. The definition of a negative sample pair also depends on the task and dataset. For example, in a face recognition task, a negative sample pair might be facial images from different people; in an image retrieval task, a negative sample pair might be images with unrelated content or significant differences. In other words, a negative sample pair refers to a sample pair with a similarity less than or equal to a preset similarity level.

[0059] 17. Model tuning refers to the process of optimizing model performance by adjusting model parameters and hyperparameters in machine learning and deep learning.

[0060] 18. The training process of a large language model includes: a pre-training phase and a fine-tuning phase.

[0061] The pre-training phase is one of the key steps in training large language models, especially for Transformer-based encoders such as BERT. During the pre-training phase, the model is trained using massive unlabeled internet text datasets through self-supervised learning.

[0062] The fine-tuning phase is performed based on the pre-trained model, with the aim of optimizing the model for a specific task. The fine-tuning phase can include a supervised fine-tuning (SFT) sub-phase.

[0063] It should be understood that SFT refers to further training a pre-trained large language model using labeled data to improve its performance on a specific task or domain. This technique leverages the parameters and structure of a pre-trained model, avoiding the need to train the model from scratch, thereby accelerating the training process and improving the model's performance on the target task.

[0064] The technical problems to be solved, the inventive concept and the system architecture of the embodiments of this application will be described below:

[0065] As mentioned above, text recall models inevitably require a large amount of training data for iterative optimization. Currently, training data is mainly constructed manually. However, for text recall models in vertical domains, the construction of training data requires higher levels of expertise from personnel. This manual approach inevitably leads to problems such as low efficiency and low accuracy in training data generation.

[0066] To address the aforementioned technical issues, this application proposes an automatic training data generation method, comprising: inputting a first document from the data source corresponding to the vertical domain text retrieval model into a first large language model to obtain a query statement corresponding to the first document; querying the query statement in the data source to retrieve N second documents; where N is an integer greater than 1; selecting M target second documents from the N second documents that constitute negative samples with the query statement; where M is a positive integer; and for each of the M target second documents, constructing a triplet training data with a partial order relation for the general text retrieval model by combining the query statement, the first document, and the target second document. This automatic training data generation method can improve the efficiency and accuracy of training data generation compared to manual methods.

[0067] In some possible implementations, the application scenarios of the embodiments of this application are as follows: Figure 1 As shown.

[0068] Figure 1 This is a schematic diagram of an application scenario involved in an embodiment of this application, such as... Figure 1 As shown, this application scenario includes electronic device 110.

[0069] In some implementations, the electronic device 110 can be any electronic device. For example, from the perspective of training and executing the text recall model, the electronic device can be a training device or an execution device, or it can be other devices; this application embodiment does not limit this. From the perspective of the device's function and purpose, the electronic device 110 can be a terminal device or a server.

[0070] In some possible implementations, the terminal device can be a desktop computer, laptop computer, tablet computer, smartphone, tablet computer, smartwatch, virtual reality (VR) device, augmented reality (AR) device, intelligent robot, etc., but is not limited to these.

[0071] In some implementations, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0072] The electronic device 110 can generate training data through the following steps:

[0073] S1: Electronic device 110 can obtain multiple first documents from the data source corresponding to the vertical domain text recall model and store them in the document library;

[0074] S2: Electronic device 110 can obtain the first document from the document library and input each first document into the first large language model to obtain the query statement corresponding to each first document;

[0075] S3: Electronic device 110 can store each query statement and the corresponding first document as a positive sample into a positive sample database;

[0076] S4: For each query statement, electronic device 110 can query the query statement in the data source;

[0077] S5: Recall multiple second documents;

[0078] S6: For each query statement, the electronic device 110 can select at least one target second document that constitutes a negative sample with the query statement from among the multiple second documents corresponding to the query statement;

[0079] S7: Electronic device 110 can store each query statement and the corresponding target second document as a negative sample into a negative sample database;

[0080] S8: For each query statement, the electronic device 110 can obtain the corresponding first document and target second document from the positive sample database and the negative sample database to form a triplet training data with a partial order relationship. The triplet training data is the training data used to train the general text recall model, or in other words, the triplet training data is the training data used to implement small model distillation training.

[0081] It should be noted that, Figure 1 This is merely a schematic diagram illustrating one application scenario provided by an embodiment of this application; the application scenarios involved in this embodiment are not limited to those described above. Figure 1 The application scenarios shown, for example, the number of electronic devices 110 is not limited to Figure 1 The one shown can be one, but there can also be multiple.

[0082] It should be understood that the training data generated using the method provided in this application can be used to train a general text retrieval model to obtain a vertical domain text retrieval model. This vertical domain text retrieval model can embed representations of queries and documents, then calculate the similarity between the two embeddings, using this similarity as the semantic matching score between the query and the document. Finally, it can retrieve documents with higher semantic matching scores. This vertical domain text retrieval model can be applied to products or scenarios such as intelligent customer service, news search, short text deduplication, short text search, and short text clustering.

[0083] For example, Figure 2A and Figure 2B This is a schematic diagram of the interface of the intelligent customer service product provided in the embodiments of this application. Users can... Figure 2A The interface shown prompts the user to input "Why hasn't my redeemed product arrived yet?" The backend server of this intelligent customer service product can use a vertical domain text retrieval model to embed the user's query and various questions in the knowledge base. It then calculates the similarity between the embedded representation of the query and the embedded representation of each question in the knowledge base, recalling the question with the highest similarity: "When will the funds arrive?", and providing the corresponding answer, such as... Figure 2B The interface shown displays the question with the highest similarity to the query, "Withdrawal arrival time," and provides the corresponding answer.

[0084] For example, in a news and information search scenario, a user can enter a query in the search box. The backend server of the search engine can use a vertical domain text retrieval model to embed the user's query and each document in the knowledge base, calculate the similarity between the embedding representation of the query and the embedding representation of each document in the knowledge base, retrieve documents with high similarity in the knowledge base, and push them to the terminal device.

[0085] For example, in the short text deduplication scenario, the backend server can obtain the target short text, perform embedding representation on the target short text and each text in the text library based on the vertical domain text recall model, calculate the similarity between the embedding representation corresponding to the target short text and the embedding representation corresponding to each text in the text library, recall the texts with high similarity in the text library, and delete these texts.

[0086] For example, in a short text search scenario, a user can enter a short text in the search box. The backend server of the search engine can embed the short text entered by the user and each text in the text library based on the vertical domain text retrieval model, calculate the similarity between the embedding representation of the short text and the embedding representation of each text in the text library, retrieve the text with high similarity in the text library, and push it to the terminal device.

[0087] For example, in a short text clustering scenario, the backend server can obtain the target short text, embed the target short text and each text in the text library based on the vertical domain text retrieval model, calculate the similarity between the embedding representation of the target short text and the embedding representation of each text in the text library, recall the texts with high similarity in the text library, and form a cluster of the target short text and the texts with high similarity with it.

[0088] The technical solution of this application will be described in detail below:

[0089] Figure 3 This is a flowchart illustrating a method for generating training data according to an embodiment of this application. This method can be executed by an electronic device, which can be any electronic device. For example, from the perspective of training and executing a text recall model, the electronic device can be a training device, an execution device, or other devices; this embodiment does not limit this. From the perspective of the device's function and purpose, the electronic device can be a terminal device or a server. Figure 3 As shown, the method may include:

[0090] S310: Input the first document from the data source corresponding to the vertical domain text retrieval model into the first language model to obtain the query statement corresponding to the first document;

[0091] In some possible implementations, if the vertical domain text recall model is a text recall model associated with a smart customer service product in a specific domain, then the data source corresponding to the vertical domain text recall model can be a knowledge base in the specific domain associated with the smart customer service product. The knowledge base stores several questions and the answers to each question. Based on this, the first document mentioned above can be any question in the knowledge base.

[0092] For example, if the vertical domain text recall model is a text recall model associated with intelligent customer service products in the banking business domain, then the data source corresponding to the vertical domain text recall model can be a knowledge base in the banking business domain, where the knowledge base stores several questions and the answers corresponding to each question.

[0093] In some implementations, if the vertical domain text recall model is a text recall model associated with a search engine in a specific domain, then the data source corresponding to the vertical domain text recall model can be a knowledge base in the specific domain associated with the search engine, wherein the knowledge base stores several documents, and based on this, the aforementioned first document can be any document in the knowledge base.

[0094] For example, if the vertical domain text retrieval model is a text retrieval model associated with a news and information search engine, then the data source corresponding to the vertical domain text retrieval model can be a knowledge base in the news and information domain, where the knowledge base stores several news and information-related documents.

[0095] In some possible implementations, if the vertical domain text recall model is a text recall model associated with a short text deduplication product in a specific domain, then the data source corresponding to the vertical domain text recall model can be a text library in the specific domain associated with the short text deduplication product. The text library stores several texts, and based on this, the first document mentioned above can be any text in the text library.

[0096] For example, if the vertical domain text recall model is a text recall model for short text deduplication and product association in the entertainment information field, then the data source corresponding to the vertical domain text recall model can be a text library in the entertainment information field, where the text library stores several texts related to entertainment information.

[0097] In some possible implementations, if the vertical domain text recall model is a text recall model associated with a short text search product in a specific domain, then the data source corresponding to the vertical domain text recall model can be a text library in the specific domain associated with the short text search product, wherein the text library stores several texts, and based on this, the aforementioned first document can be any text in the text library.

[0098] For example, if a vertical domain text recall model is a text recall model associated with a short text search product in the field of sports information, then the data source corresponding to this vertical domain text recall model can be a text library in the field of sports information, wherein the text library stores a number of texts related to sports information.

[0099] In some possible implementations, if the vertical domain text recall model is a text recall model associated with a short text clustering product in a specific domain, then the data source corresponding to the vertical domain text recall model can be a text library in the specific domain associated with the short text clustering product, wherein the text library stores several texts, and based on this, the aforementioned first document can be any text in the text library.

[0100] For example, if a vertical domain text recall model is a text recall model associated with short text clustering products in the parenting information field, then the data source corresponding to this vertical domain text recall model can be a text library in the parenting information field, where the text library stores several texts related to parenting information.

[0101] In some implementations, the data source corresponding to the vertical domain text retrieval model can also be a database that stores session records. Based on this, the first document mentioned above can be any session record in the database.

[0102] It should be understood that the embodiments of this application do not limit the content in the data source corresponding to the vertical domain text recall model.

[0103] It should be understood that the query statement corresponding to the first document can be understood as a query statement related to the first document, or as a query statement with a similarity or relevance to the first document that is higher than the similarity threshold, but is not limited to this.

[0104] In some implementations, the primary language model can be trained using training samples. Each training sample can include a document and its corresponding query statement, where the query statement is the actual query statement corresponding to the document and can serve as a sample label. The training device can employ supervised training methods; for example, it can input the document into the primary language model and output the predicted query statement corresponding to the document. Further, the training device can calculate a loss based on the actual query statements and their corresponding predicted query statements included in all training samples, and adjust the parameters of the primary language model based on this loss until the training iterations reach a preset number or the loss reaches its minimum value, at which point training stops.

[0105] In some implementations, the training device may use any of the following loss functions when training the primary language model, but is not limited to: L1 loss function, mean squared error (MSE) loss function, cross-entropy loss function, etc.

[0106] In some implementations, when an electronic device inputs a first document into a first language model, it can input the document according to a preset input format to output a query statement that conforms to the preset output format.

[0107] For example, the default input format of the first major language model is as follows:

[0108] Please generate the relevant query statements based on the following document:

[0109] {{document}}

[0110] Query statement:

[0111] The default output format of the first major language model is as follows:

[0112] {{Query Statement}}

[0113] It should be understood that the embodiments of this application do not limit the preset input format and preset output format of the first language model.

[0114] In some implementations, before the electronic device inputs the first document from the data source corresponding to the vertical domain text retrieval model into the first large language model to obtain the query statement corresponding to the first document, the method further includes: the electronic device preprocessing the first document; correspondingly, the electronic device inputting the first document from the data source corresponding to the vertical domain text retrieval model into the first large language model to obtain the query statement corresponding to the first document includes: the electronic device inputting the preprocessed first document into the first large language model to obtain the query statement corresponding to the first document.

[0115] In some possible implementations, the electronic device preprocesses the first document, including, but not limited to, at least one of the following:

[0116] Replace specific characters in the first document with placeholders;

[0117] Remove duplicate content from the first document;

[0118] Filter out noise in the first document.

[0119] It should be understood that the embodiments of this application can protect privacy information by replacing specific characters in the first document with placeholders.

[0120] In some feasible ways, electronic devices can identify specific characters in the first document using regular expressions.

[0121] In some implementations, the electronic device can compare a first document with specific characters in a specific character set to identify a specific character in the first document.

[0122] It should be understood that the embodiments of this application do not limit the recognition method of specific characters.

[0123] In some implementations, all specific characters in the first document can correspond to the same placeholder. For example, the Uniform Resource Locator (URL) and the phone number in the first document can both be replaced with the placeholder 0.

[0124] In some implementations, there is a one-to-one correspondence between specific characters and placeholders. For example, the placeholder for the URL in the first document is... <url>The placeholder for the phone number is<phone number> .

[0125] In some implementations, the first document may contain multiple instances of a specific character corresponding to the same placeholder, or a single instance of a specific character corresponding to a different placeholder. For example, the placeholder corresponding to the Uniform Resource Locator (URL) in the first document might be... <url>The placeholders for both the phone number and home address are 0.

[0126] It should be understood that the embodiments of this application do not limit the correspondence between specific characters and placeholders.

[0127] It should be understood that, by deduplicating the duplicate content in the first document, the embodiments of this application can reduce the length of the input data of the first language model, thereby improving the data processing efficiency of the model.

[0128] In some possible implementations, the electronic device deduplicates content in the first document by: the electronic device acquiring each statement in the first document, then calculating the hash value of each statement, and retaining only one statement for statements with the same hash value.

[0129] For example, if the hash values ​​of statements 1 and 2 in the first document are both 10, it means that statements 1 and 2 are the same or similar. Based on this, the electronic device can retain only statements 1 or 2 in the first document.

[0130] In some possible implementations, the electronic device deduplicates duplicate content in the first document, including: the electronic device can obtain each sentence in the first document, then calculate the similarity between any two sentences, and define sentences with a similarity greater than a similarity threshold as similar sentences; furthermore, the electronic device retains only one sentence among the similar sentences.

[0131] For example, if the similarity between statement 1 and statement 2 in the first document is greater than the similarity threshold of 70%, and the similarity between statement 1 and statement 3 is also greater than the similarity threshold of 70%, then it can be determined that statements 1, 2, and 3 are similar statements, and the electronic device can retain only statement 1, statement 2, or statement 3 from the first document.

[0132] It should be understood that the embodiments of this application do not limit the method of deduplication of document content.

[0133] It should be understood that by filtering noise in the first document, the embodiments of this application can improve the quality of the first document, thereby helping the first model to output more accurate query statements.

[0134] In some implementations, the electronic device filters noise in the first document, including at least one of the following: removing misspelled words; removing incomplete sentences.

[0135] In some implementations, the electronic device can input a first document into a noise filtering model to obtain a noise-filtered document.

[0136] In some implementations, the noise filtering model can be trained using training samples. Each training sample can include a document and an actual document after noise filtering, where the actual document after noise filtering can serve as the sample label. The training device can employ supervised training; for example, it can input the document into the noise filtering model and output a predicted document after noise filtering. Further, the training device can calculate a loss based on all the actual and predicted documents included in the training samples, and adjust the parameters of the noise filtering model based on this loss until a preset number of training iterations is reached, or the loss reaches its minimum value, at which point training stops.

[0137] In some implementations, the training device may use any of the following loss functions when training the noise filtering model, but is not limited to: L1 loss function, MSE loss function, cross-entropy loss function, etc.

[0138] It should be understood that the embodiments of this application do not limit the noise reduction method for documents.

[0139] S320: Query the data source to retrieve N second documents; where N is an integer greater than 1.

[0140] S330: Select M target second documents from N second documents that form a negative sample with the query statement; where M is a positive integer;

[0141] The following is a combined explanation of S320 and S330:

[0142] In some implementations, the electronic device queries the data source for a query statement to retrieve N second documents, including: the electronic device queries the data source for the query statement and retrieves the first N second documents in descending order of similarity between the query statement and the documents in the data source.

[0143] For example, an electronic device can query a data source for a certain query statement and retrieve the top 250 second documents in descending order of similarity between the query statement and the documents in the data source.

[0144] In some implementations, the electronic device queries a data source for a query statement and retrieves the top N second documents based on their similarity to the documents in the data source, ranked from highest to lowest. This includes the electronic device inputting the query statement and the documents in the data source into a third language model to retrieve the top N second documents based on their similarity to the documents in the data source, ranked from highest to lowest. In other words, the electronic device uses this third language model to calculate the order of similarity between the query statement and the documents in the data source, ranked from highest to lowest, and retrieves the top N second documents.

[0145] In some implementations, the electronic device queries a query statement in a data source to retrieve N second documents, including: the electronic device queries the query statement in the data source, retrieves the first R second documents in descending order of similarity to the query statement, and then retrieves N second documents from the first R second documents, where R is an integer greater than N.

[0146] In some implementations, the electronic device queries a data source for a query statement and retrieves the top R second documents based on their similarity to the documents in the data source, in descending order. This includes: the electronic device inputs the query statement and the documents in the data source into a third language model to retrieve the top R second documents based on their similarity to the documents in the data source, in descending order. In other words, the electronic device can use this third language model to calculate the order of similarity between the query statement and the documents in the data source, in descending order, and retrieve the top R second documents.

[0147] In some possible implementations, the electronic device recalls N second documents from the first R second documents, including: the electronic device randomly selects N second documents from the first R second documents.

[0148] For example, an electronic device can query a query statement in a data source, and retrieve the first 300 second documents in descending order of similarity to the query statement, and then randomly select 250 second documents from these 300 second documents.

[0149] In some possible implementations, the electronic device recalls N second documents from the first R second documents, including: the electronic device recalls one second document every Q second documents from the first R second documents, thereby obtaining N second documents.

[0150] For example, an electronic device can query a query statement in a data source and retrieve the first 1,000 second documents in descending order of similarity to the query statement. Then, starting from the first second document, it retrieves one second document at a time, stopping the retrieval process when 250 second documents have been retrieved.

[0151] In some possible implementations, the electronic device selects M target second documents from N second documents that constitute negative samples with the query statement, including: the electronic device selects M second documents from the Pth second document to the Nth second document in descending order of similarity between the query statement and the N second documents, as the M target second documents.

[0152] In some possible implementations, the electronic device selects M second documents from the Pth to the Nth second documents, including: the electronic device randomly selects M second documents from the Pth to the Nth second documents.

[0153] For example, an electronic device can query a query statement in a data source, retrieve 250 second documents in descending order of similarity between the documents in the data source and the query statement, and then the electronic device can randomly select 7 second documents from the 50th to the 250th second documents.

[0154] In some possible implementations, the electronic device selects M second documents from the Pth to the Nth second documents, including: the electronic device selects the last M second documents from the Pth to the Nth second documents.

[0155] For example, an electronic device can query a query statement in a data source, retrieve 250 second documents in descending order of similarity between the documents in the data source and the query statement, and then the electronic device can select the 244th to 250th second documents from the 50th to the 250th second documents.

[0156] In some possible implementations, the electronic device selects M target second documents from N second documents that constitute negative samples with the query statement, including: the electronic device selects the last M second documents from the N second documents in descending order of similarity between the query statement and the N second documents, as the M target second documents.

[0157] For example, an electronic device can query a query statement in a data source and retrieve 250 second documents. Then, the electronic device can select the 244th to 250th second documents in descending order of similarity between the query statement and these 250 second documents.

[0158] S340: For each of the M target second documents, training data with a partial order relation consisting of a query statement, a first document, and a target second document to form a general text recall model.

[0159] It should be understood that all the first documents and target second documents corresponding to the same query statement can constitute a document set. According to the definition of a partially ordered set, this document set is a partially ordered set, meaning that some of the documents contained therein have a partial order relationship. For example, the first document and target second document corresponding to the same query statement have a partial order relationship. Since the query statement corresponding to the first document can be understood as a query statement related to the first document, or as a query statement whose similarity or relevance to the first document is higher than a similarity threshold, it can be determined that the sample <query statement, first document> is a positive sample, while the sample <first document, target second document> is a negative sample. Based on this, it can be determined that the partial order relationship between the first document and the target second document is target second document ≤ first document.

[0160] In this embodiment, the reason for proposing training data with a partial order relation of triples is that the contrastive loss function can be used to calculate the contrastive loss of a general text recall model. In the process of calculating the contrastive loss, training data with a partial order relation of triples can clearly define the order between documents, so as to better calculate the contrastive loss of the model.

[0161] It should be understood that since the sample <query statement, first document> is a positive sample and the sample <first document, target second document> is a negative sample, these two samples can form a negative sample pair. Based on this, the label of the triple training data <query statement, first document, target second document> consisting of the query statement, the first document, and the target second document can be 0, indicating that the triple training data is a negative sample pair.

[0162] It should be understood that, since the triple training data <query statement, first document, target second document> consisting of the query statement, the first document, and the target second document includes the first document and the target second document with a partial order relation, the triple training data can be referred to as triple training data with a partial order relation, or it can also be referred to as partial order training data, partial order triple, or partial order triple training data, etc., and the embodiments of this application do not limit this.

[0163] In some implementations, the electronic device can also sort the M target second documents based on the similarity between the query statement and the M target second documents; for any two target second documents among the M target second documents, the query statement and the two target second documents are used to form training data of triples with a partial order relationship for a general text recall model, and based on the order of the M target second documents, labels are generated for the training data of triples corresponding to the two target second documents; wherein, the labels are used to identify whether the training data of triples corresponding to the two target second documents are positive sample pairs or negative sample pairs.

[0164] In some implementations, the electronic device ranks the M target second documents based on the similarity between the query and the M target second documents. This includes: the electronic device inputs the M target second documents and the query into a second large language model to rank the M target second documents based on the similarity between the query and the M target second documents. In other words, the electronic device can rank the M target second documents based on the similarity between the query and the M target second documents using this second large language model.

[0165] In some feasible implementations, when an electronic device inputs M target second documents into a second large language model, it can input the documents according to a preset input format to output sorting results.

[0166] For example, the default input format for the second largest language model is as follows:

[0167] The following is a series of documents related to {{query statements}}:

[0168] [1]{{Document_1}}

[0169] [2]{{Document_2}}

[0170] [3]{{Document_3}}

[0171] [4]{{Document_4}}

[0172] [5]{{Document_5}}

[0173] [6]{{Document_6}}

[0174] [7]{{Document_7}}

[0175] Please sort the above documents according to their relevance to the query:

[0176] Sorting results:

[0177] [5]>[2]>[1]>[4]>[3]>[7]>[6]

[0178] It should be understood that the embodiments of this application do not limit the preset input format of the second language model.

[0179] It should be understood that electronic devices can determine the partial order relationship between M target second documents based on the order of the M target second documents. For example, suppose there are 7 target second documents and their sorting result is [5]>[2]>[1]>[4]>[3]>[7]>[6]. Based on this, the partial order relationship between the 7 target second documents can be obtained as: document_6≤document_7≤document_3≤document_4≤document_1≤document_2≤document_5.

[0180] In some implementations, based on the order of the M target second documents, labels are generated for the training data of triples corresponding to any two target second documents, including: if the difference in rank between any two target second documents is greater than a preset threshold, then labels are generated to identify that the training data of triples corresponding to any two target second documents are negative sample pairs.

[0181] For example, assuming the preset threshold is 3, if the ranking difference between target second document 1 and target second document 2 is 5, which is greater than the preset threshold of 3, then the sample pair consisting of sample <query statement, target second document 1> and another sample <query statement, target second document 2> can be determined as a negative sample pair.

[0182] In some possible implementations, based on the order of the M target second documents, labels are generated for the triplet training data corresponding to any two target second documents, including: if the rank difference between any two target second documents is less than or equal to a preset threshold, then labels are generated to identify that the triplet training data corresponding to any two target second documents are positive sample pairs.

[0183] For example, assuming the preset threshold is 3, if the ranking difference between target second document 1 and target second document 3 is 1, which is less than the preset threshold 3, then the sample pair consisting of sample <query statement, target second document 1> and another sample <query statement, target second document 3> can be determined as a negative sample pair.

[0184] It should be understood that, since the training data of the triples consisting of the query statement and any two target second documents includes any two target second documents with a partial order relation, the training data of the triples can be called training data of triples with a partial order relation, or it can be called partial order training data, partial order triples, or partial order triple training data, etc., and the embodiments of this application do not limit this.

[0185] In some feasible implementations, electronic devices can also train a general text recall model based on training data of triples with partial order relations of a general text recall model to obtain a vertical domain text recall model.

[0186] In some implementations, electronic devices can calculate the contrastive loss of a general text recall model based on training data of triples with partial order relations, and then train the general text recall model based on the contrastive loss to obtain a vertical domain text recall model.

[0187] The contrastive loss function can be as follows:

[0188]

[0189] Where L represents the contrastive loss, and M is the number of target second documents. It is the number of sample pairs, y i D is the label corresponding to the i-th sample pair. A value of 1 indicates a positive sample pair, and a value of 0 indicates a negative sample pair. i represents the distance between the i-th sample pair, and margin is a hyperparameter representing the minimum distance between the two samples included in a negative sample pair.

[0190] It should be understood that the process by which an electronic device trains a general text retrieval model to obtain a vertical domain text retrieval model can be referred to as a model tuning process, the SFT stage, or a small model distillation training process, etc. This application's embodiments do not limit this; the following will be combined with... Figure 4 The training process for the small-scale distillation model is explained below:

[0191] Figure 4 A schematic diagram of a small-model distillation training process provided in an embodiment of this application is shown below. Figure 4 As shown, the general text retrieval model is logically a twin network, consisting of two identical networks, one for processing queries and the other for processing documents. The query processing can include: segmenting the query into words to obtain multiple tokens, and then encoding these tokens using a query encoder to obtain the corresponding embedding vector. Similarly, the document processing can include: segmenting the document into words to obtain multiple tokens, and then encoding these tokens using a document encoder to obtain the corresponding embedding vector; finally, the similarity between the embedding vectors of the query and the document is calculated, and this similarity can be denoted as the <query, document> similarity.

[0192] Furthermore, electronic devices can train a general text recall model using training data consisting of multiple triples with partial order relationships, such as... Figure 4 As shown, suppose there are M target second documents, namely target second document 1, target second document 2, ..., target second document M. Suppose the partial order relation of the M target second documents is target second document 1' ≤ target second document 2' ... ≤ target second document M'. Then these triple training data include: <query statement, first document, target second document 1'>, <query statement, first document, target second document 2'> ... <query statement, first document, target second document M'>, <query statement, target second document M', target second document (M-1)'>, <query statement, target second document (M-1)', target second document (M-2)'> ... <query statement, target second document 2', target second document 1'>. For any triple training data, the two networks of the Siamese network are used to process the query statement and the document, respectively. The document includes: the first document and any target second document, or the document includes: any two target second documents. The processing of a query statement can include: segmenting the query statement into words to obtain multiple word segments, and then encoding these word segments using a query statement encoder to obtain the corresponding embedding vector. Similarly, the processing of a document can include: segmenting the document into words to obtain multiple word segments, and then encoding these word segments using a document encoder to obtain the corresponding embedding vector; finally, the similarity between the embedding vectors corresponding to the query statement and the document can be calculated, and this similarity can be denoted as the similarity between <query statement, document>. Furthermore, for any triple training data, after obtaining two similarity scores, the electronic device can calculate the distance between the sample pairs included in the triple training data based on these two similarity scores, and then calculate the contrastive loss of a general text recall model based on this distance.

[0193] It should be understood that, Figure 4 In the text, "document+" and "document-" represent the first and last ranked documents in a partial order relation, respectively.

[0194] Finally, the model is optimized using the contrastive loss of a general text recall model to obtain a text recall model for the vertical domain.

[0195] This application provides a method for generating training data, comprising: inputting a first document from a data source corresponding to a vertical domain text retrieval model into a first large language model to obtain a query statement corresponding to the first document; querying the query statement in the data source to retrieve N second documents; wherein N is an integer greater than 1; selecting M target second documents from the N second documents that constitute negative samples with the query statement; wherein M is a positive integer; and for each of the M target second documents, constructing a triplet training data with a partial order relation for a general text retrieval model by combining the query statement, the first document, and the target second document. Since the query statement, the first document, and the target second documents are all automatically obtained, automated generation of training data is achieved. This automated training data generation method can improve the efficiency and accuracy of training data generation compared to manual methods.

[0196] Furthermore, in this embodiment, the M target second documents can be sorted, and based on their order, training data of triples with a partial order relationship, such as <query statement, target second document, target second document>, can be constructed. On the one hand, this increases the amount of training data, which is helpful for training the text recall model. On the other hand, since the target second document and the query statement constitute negative samples, this method can improve the utilization rate of negative samples.

[0197] Furthermore, by sorting the M target second documents, this embodiment can distinguish whether the training data consisting of a query statement and any two target second documents are positive or negative sample pairs, thereby achieving automatic labeling of sample pairs and improving sample labeling efficiency. Experiments have shown that, assuming the same batch of samples needs to be labeled, manual labeling can only complete about 100 to 150 sample labels per day; while the automatic labeling method provided in this embodiment can label more than 40,000 sample labels per day.

[0198] The following is combined Figure 5 The process of generating training data is illustrated by example:

[0199] For example, Figure 5 This is a schematic diagram illustrating a training data generation process provided in an embodiment of this application, such as... Figure 5 As shown, the process of generating training data includes:

[0200] S501: Electronic devices can obtain multiple first documents from the data source corresponding to the vertical domain text retrieval model and store them in the document library;

[0201] S502: The electronic device can retrieve the first document from the document library and input each first document into the first large language model to obtain the query statement corresponding to each first document;

[0202] S503: The electronic device can store each query statement and the corresponding first document as a positive sample in the positive sample database;

[0203] S504: For each query statement, the electronic device can query the query statement in the data source;

[0204] S505: Recall multiple second documents;

[0205] S506: For each query statement, the electronic device may select at least one target second document that constitutes a negative sample with the query statement from among a plurality of second documents corresponding to the query statement;

[0206] S507: Electronic devices may store each query statement and the corresponding target second document as a negative sample in a negative sample database;

[0207] S508: For each query, the electronic device can obtain the corresponding first document and target second document from the positive sample database and the negative sample database to form training data of triples with partial order relation;

[0208] S509: The electronic device stores the above triplet training data into the triplet library;

[0209] S510: The electronic device sorts the seven target second documents P1, P2...P7 based on the second largest language model;

[0210] S511: For electronic devices, any two target second documents out of seven target second documents; the query statement and any two target second documents constitute the training data of a triple with a partial order relation for a general text recall model;

[0211] S512: The electronic device stores the above triplet training data into the triplet library.

[0212] This example demonstrates that the query statement, the first document, and the target second document are all automatically generated, achieving automated generation of training data. Compared to manual methods, this automatic training data generation method can improve the efficiency and accuracy of training data generation.

[0213] The following is combined Figure 6 The model training process is illustrated by example:

[0214] For example, Figure 6 This is a schematic diagram of a model training process provided in an embodiment of this application, as shown below. Figure 6 As shown, the process of generating training data includes:

[0215] S601: Electronic devices can obtain multiple first documents from the data source corresponding to the vertical domain text retrieval model and preprocess each first document;

[0216] It should be understood that the preprocessing procedure for the first document can be referred to above, and will not be repeated in this embodiment.

[0217] S602: For each first document, the electronic device can obtain the query statement corresponding to that first document;

[0218] In some possible implementations, for each first document, the electronic device can input the first document into the first language model to obtain the query statement corresponding to the first document.

[0219] S603: The electronic device can store each query statement and the corresponding first document as a positive sample in the positive sample database;

[0220] S604: For each query statement, the electronic device can query the query statement in the data source to retrieve multiple second documents;

[0221] S605: For each query statement, the electronic device may select multiple target second documents that constitute a negative sample with the query statement from multiple second documents corresponding to the query statement;

[0222] S606: Electronic devices may store each query statement and the corresponding target second document as a negative sample in a negative sample database;

[0223] S607: For each query, the electronic device can obtain the corresponding first document and target second document from the positive sample database and the negative sample database to form training data of triples with partial order relation;

[0224] S608: The electronic device sorts the target second document;

[0225] S609: For electronic devices targeting any two target second documents from multiple target second documents; the query statement and any two target second documents constitute the training data of a triple with a partial order relation for a general text recall model;

[0226] S610: The electronic device stores the triplet training data into the triplet library;

[0227] S611: Electronic devices train a general text recall model using triple training data from a triple library to obtain a vertical domain text recall model.

[0228] This example demonstrates that the query statement, the first document, and the target second document are all automatically generated, achieving automated generation of training data. Compared to manual methods, this automatic training data generation method can improve the efficiency and accuracy of training data generation, thereby improving the efficiency and accuracy of model training.

[0229] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0230] It should also be understood that, in the various method embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0231] The method provided in the embodiments of this application has been described above. The training data generation apparatus provided in the embodiments of this application will be described below.

[0232] Figure 7 A schematic diagram of a training data generation apparatus 700 provided in an embodiment of this application is shown below. Figure 7 As shown, the device 700 includes: an input module 710, a recall module 720, a selection module 730, and a generation module 740. The input module 710 inputs a first document from the data source corresponding to the vertical domain text recall model into the first large language model to obtain the query statement corresponding to the first document. The recall module 720 queries the query statement in the data source to recall N second documents, where N is an integer greater than 1. The selection module 730 selects M target second documents from the N second documents that constitute negative samples with the query statement, where M is a positive integer. The generation module 740, for each of the M target second documents, constructs a triplet training data with a partial order relation for the general text recall model, consisting of the query statement, the first document, and the target second document.

[0233] In some implementations, the apparatus 700 further includes: a sorting module 750, used to sort the M target second documents based on the similarity between the query statement and the M target second documents; and a generation module 740, used to: for any two target second documents among the M target second documents, construct a triplet training data with a partial order relationship for a general text recall model using the query statement and the arbitrary two target second documents, and generate labels for the triplet training data corresponding to the arbitrary two target second documents based on the order of the M target second documents; wherein the labels are used to identify whether the triplet training data corresponding to the arbitrary two target second documents is a positive sample pair or a negative sample pair.

[0234] In some implementations, the generation module 740 is specifically used to: generate labels to identify that the training data of the triplets corresponding to any two target second documents are negative sample pairs if the difference in ranking between any two target second documents is greater than a preset threshold; or, generate labels to identify that the training data of the triplets corresponding to any two target second documents are positive sample pairs if the difference in ranking between any two target second documents is less than or equal to the preset threshold.

[0235] In some implementations, the sorting module 750 is specifically used to: input M target second documents and query statements into a second large language model, and sort the M target second documents based on the similarity between the query statements and the M target second documents.

[0236] In some implementations, the recall module 720 is specifically used to: query the query statement in the data source and recall the first N second documents in descending order of similarity between the query statement and the documents in the data source.

[0237] In some implementations, the recall module 720 is specifically used to: input the query statement and the documents in the data source into the third language model, so as to recall the first N second documents in descending order of similarity between the query statement and the documents in the data source.

[0238] In some implementations, the selection module 730 is specifically used to: select M second documents from the Pth to the Nth second documents in descending order of similarity between the query statement and the N second documents, as the M target second documents; or, select the last M second documents from the N second documents in descending order of similarity between the query statement and the N second documents, as the M target second documents.

[0239] In some implementations, the selection module 730 is specifically used to: randomly select M second documents from the Pth to the Nth second documents.

[0240] In some implementations, the selection module 730 is specifically used to select the last M second documents from the Pth to the Nth second documents.

[0241] In some implementations, the apparatus 700 further includes a training module 760 for training the general text recall model on training data of triples with partial order relations based on the general text recall model to obtain a vertical domain text recall model.

[0242] In some implementations, the apparatus 700 further includes a preprocessing module 770, used to preprocess the first document before the input module 710 inputs the first document from the data source corresponding to the vertical domain text retrieval model into the first large language model to obtain the query statement corresponding to the first document; accordingly, the preprocessing module 770 is specifically used to input the preprocessed first document into the first large language model to obtain the query statement.

[0243] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 7 The device 700 shown can perform Figure 3 The corresponding method embodiments, and the foregoing and other operations and / or functions of each module in device 700 are respectively implemented to achieve Figure 3 For the sake of brevity, the corresponding processes in each method are not described in detail here.

[0244] The apparatus 700 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0245] Figure 8 This is a schematic block diagram of the electronic device 800 provided in an embodiment of this application. Figure 8 As shown, the electronic device 800 may include:

[0246] The system includes a memory 810 and a processor 820. The memory 810 stores a computer program 830 and transfers the computer program 830 to the processor 820. In other words, the processor 820 can retrieve and run the computer program 830 from the memory 810 to implement the methods described in the embodiments of this application.

[0247] For example, the processor 820 can be used to execute the steps in the above method according to the instructions in the computer program 830.

[0248] In some embodiments of this application, the processor 820 may include, but is not limited to:

[0249] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0250] In some embodiments of this application, the memory 810 includes, but is not limited to:

[0251] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0252] In some embodiments of this application, the computer program 830 may be divided into one or more modules, which are stored in the memory 810 and executed by the processor 820 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 830 in the electronic device.

[0253] like Figure 8 As shown, the electronic device 800 may further include:

[0254] Transceiver 840, which can be connected to processor 820 or memory 810.

[0255] The processor 820 can control the transceiver 840 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 840 may include a transmitter and a receiver. The transceiver 840 may further include antennas, and the number of antennas may be one or more.

[0256] It should be understood that the various components in the electronic device 800 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0257] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0258] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above-described method embodiments.

[0259] In other words, when implemented using software, it can be implemented wholly or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0260] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0261] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0262] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0263] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / url> < / url>

Claims

1. A method for generating training data, characterized in that, include: Input the first document from the data source corresponding to the vertical domain text retrieval model into the first large language model to obtain the query statement corresponding to the first document. The query statement is queried in the data source to retrieve N second documents; where N is an integer greater than 1. Select M target second documents from the N second documents that constitute negative samples with the query statement; where M is a positive integer; For each of the M target second documents, the query statement, the first document, and the target second document constitute the training data of a triple with a partial order relation for a general text recall model.

2. The method according to claim 1, characterized in that, Also includes: Based on the similarity between the query statement and the M target second documents, the M target second documents are sorted. For any two target second documents among the M target second documents, the query statement and the two target second documents constitute the training data of the general text recall model with a partial order relationship of triples, and based on the order of the M target second documents, the labels of the training data of the triples corresponding to the two target second documents are generated. The label is used to identify whether the training data of the triplet corresponding to any two target second documents is a positive sample pair or a negative sample pair.

3. The method according to claim 2, characterized in that, The step of generating labels for the triplet training data corresponding to any two target second documents based on the order of the M target second documents includes: If the difference in ranking between any two target second documents is greater than a preset threshold, then a label is generated to identify that the triplet training data corresponding to any two target second documents is a negative sample pair. If the ranking difference between any two target second documents is less than or equal to a preset threshold, then a label is generated to identify that the triplet training data corresponding to any two target second documents is a positive sample pair.

4. The method according to claim 2, characterized in that, The step of ranking the M target second documents based on the similarity between the query statement and the M target second documents includes: The M target second documents and the query statement are input into the second large language model to sort the M target second documents based on the similarity between the query statement and the M target second documents.

5. The method according to any one of claims 1-4, characterized in that, The step of querying the query statement in the data source to retrieve N second documents includes: The query statement is queried in the data source, and the first N second documents are retrieved in descending order of similarity between the query statement and the documents in the data source.

6. The method according to claim 5, characterized in that, The step involves querying the query statement in the data source and retrieving the top N second documents in descending order of similarity between the query statement and the documents in the data source, including: The query statement and the documents in the data source are input into a third language model to retrieve the top N second documents in descending order of similarity between the query statement and the documents in the data source.

7. The method according to any one of claims 1-4, characterized in that, The step of selecting M target second documents from the N second documents that constitute negative samples with the query statement includes: Based on the order of similarity between the query statement and the N second documents from high to low, select M second documents from the Pth to the Nth second documents as the M target second documents; or, Based on the order of similarity between the query statement and the N second documents from highest to lowest, the last M second documents are selected as the M target second documents.

8. The method according to claim 7, characterized in that, The selection of M second documents from the Pth to the Nth second documents includes: Randomly select M second documents from the Pth to the Nth second documents; or, Select the last M second documents from the Pth to the Nth second documents.

9. The method according to any one of claims 1-4, characterized in that, Also includes: The general text recall model is trained using training data of triples with partial order relations to obtain the vertical domain text recall model.

10. The method according to any one of claims 1-4, characterized in that, Before inputting the first document from the data source corresponding to the vertical domain text retrieval model into the first large language model to obtain the query statement corresponding to the first document, the method further includes: The first document is preprocessed; The step of inputting the first document from the data source corresponding to the vertical domain text retrieval model into the first large language model to obtain the query statement corresponding to the first document includes: The preprocessed first document is input into the first large language model to obtain the query statement.

11. A training data generation apparatus, characterized in that, include: The input module is used to input the first document from the data source corresponding to the vertical domain text retrieval model into the first large language model to obtain the query statement corresponding to the first document. The recall module is used to query the query statement in the data source to recall N second documents; where N is an integer greater than 1. The selection module is used to select M target second documents from the N second documents that constitute negative samples with the query statement; where M is a positive integer; The generation module is used to generate training data for a general text recall model by constructing a triplet with a partial order relationship for each of the M target second documents, consisting of the query statement, the first document, and the target second document.

12. An electronic device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method of any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 10.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1 to 10.

Images