Question and answer text data processing method and device, electronic equipment and medium
By using a large language model to process the problem text sequence and generate the answer text sequence, and combining positive and negative samples to train the vector search model, the problem of inaccurate response of the existing model is solved, and the accuracy and efficiency of the search results are improved.
Patent Information
- Application Number
- CN202411311119.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2025-03-21
AI Technical Summary
Existing vector search models may not be able to respond effectively to input content, resulting in inaccurate or incomplete query results.
The problem text sequence is recognized and processed through a large language model, the answer text sequence is generated, and the vector search model is trained based on the positive and negative samples problem text sequences, and the model performance is optimized by loop iteration.
The relevance and accuracy of the search results are improved, the model's adaptability and robustness to different types of problems are enhanced, the proportion of error searches is reduced, and the search speed and efficiency are improved.
Smart Images
Figure CN119669394A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of question and answer text data processing, and in particular to a question and answer text data processing method, device, electronic device and medium. Background Art
[0002] According to business needs, various document data can be configured for manual review by business-related personnel. The management and retrieval of these document data usually require manual intervention, which limits the efficiency and accuracy of information retrieval. With the rapid development of Internet technology, vector retrieval models based on document data have gradually become an effective way of information retrieval.
[0003] By utilizing Internet technology, efficient vector retrieval models can be trained. These models can convert document data into vector representations and calculate the similarity between documents in vector space. Business-related personnel can access these vector retrieval models through applications or web pages, and quickly query the document library by inputting content, thereby improving the efficiency and accuracy of information retrieval. However, in actual applications, the inventors found that there are some problems with existing vector retrieval models. First, the existing models may not be able to effectively respond to the input content, resulting in inaccurate or incomplete query results. Second, the results returned by the existing models are less relevant to the actual answers, which affects the user's acquisition and utilization efficiency of information. Summary of the invention
[0004] In view of this, the embodiments of the present disclosure provide a question and answer text data processing method, device, electronic device and computer-readable storage medium to solve the technical problem in the prior art that the existing model may not be able to effectively respond to the input content, resulting in inaccurate or incomplete query results.
[0005] In a first aspect of an embodiment of the present disclosure, a method for processing question and answer text data is provided, including: obtaining a set of question text sequences; identifying and processing each question text sequence in the set of question text sequences through a large language model to obtain an answer text sequence corresponding to each question text sequence; determining a positive sample question text sequence corresponding to each question text sequence based on each question text sequence and an answer text sequence corresponding to each question text sequence; generating a corresponding negative sample question text sequence based on the positive sample question text sequence corresponding to each question text sequence; and training a vector retrieval model based on the positive sample question text sequence and each negative sample question text sequence corresponding to each question text sequence in a loop iterative manner.
[0006] According to a second aspect of an embodiment of the present disclosure, a question-answer text data processing device is provided, comprising: an acquisition module for acquiring a set of question text sequences; a recognition processing module for performing recognition processing on each question text sequence in the set of question text sequences through a large language model to obtain an answer text sequence corresponding to each question text sequence; a determination module for determining a positive sample question text sequence corresponding to each question text sequence based on each question text sequence and an answer text sequence corresponding to each question text sequence; a generation module for generating a corresponding negative sample question text sequence based on the positive sample question text sequence corresponding to each question text sequence; and a training module for training a vector retrieval model based on the positive sample question text sequence and each negative sample question text sequence corresponding to each question text sequence in a cyclic iteration manner.
[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0009] Compared with the prior art, the embodiments of the present disclosure have the following beneficial effects: the embodiments of the present disclosure can more accurately understand and process the intent of user queries by using a large language model to recognize and process question text sequences and generate corresponding answer text sequences. By accurately matching questions and answers, the relevance and accuracy of retrieval results are improved. By generating positive sample question text sequences and corresponding negative sample question text sequences, richer and more diverse training data is provided for the vector retrieval model. This method not only enhances the model's adaptability to different types of questions, but also improves the robustness of the model in practical applications. By adopting a cyclic iterative training method, the vector retrieval model can better optimize its performance after multiple rounds of training, effectively reducing the proportion of erroneous retrievals and improving retrieval speed and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0011] Figure 1A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied;
[0012] Figure 2 It is a flowchart of a method for processing question and answer text data provided by an embodiment of the present disclosure;
[0013] Figure 3 It is a flowchart of another question-answer text data processing method provided by an embodiment of the present disclosure;
[0014] Figure 4 is a block diagram of a question-answer text data processing device provided by an embodiment of the present disclosure;
[0015] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0016] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.
[0017] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.
[0018] Figure 1 FIG. 1 is a schematic diagram showing an exemplary system architecture to which the technical solution of the embodiment of the present invention can be applied. Figure 1 As shown, the system architecture 100 may include one or more of a first terminal device 101, a second terminal device 102, and a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.
[0019] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.
[0020] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send data, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, and the like.
[0021] The server 105 may be a server that provides various services. For example, the server 105 may obtain a question text sequence set from the first terminal device 103 (or the second terminal device 102 or the third terminal device 103); identify and process each question text sequence in the question text sequence set through a large language model to obtain an answer text sequence corresponding to each question text sequence; determine a positive sample question text sequence corresponding to each question text sequence based on each question text sequence and an answer text sequence corresponding to each question text sequence; generate a corresponding negative sample question text sequence based on the positive sample question text sequence corresponding to each question text sequence; and train a vector retrieval model based on the positive sample question text sequence corresponding to each question text sequence and each negative sample question text sequence in a loop iteration manner.
[0022] In some embodiments, the question-answer text data processing method provided in the embodiments of the present invention is generally executed by the server 105, and accordingly, the question-answer text data processing device is generally arranged in the server 105. In other embodiments, some terminal devices may have functions similar to those of the server to execute the method. Therefore, the question-answer text data processing method provided in the embodiments of the present invention is not limited to being executed on the server side.
[0023] The question and answer text data processing method and device according to the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
[0024] Figure 2 is a flowchart of a method for processing question and answer text data provided by an embodiment of the present disclosure. The method provided by an embodiment of the present disclosure can be executed by any electronic device with computer processing capabilities, for example, the electronic device can be Figure 1 The server is shown.
[0025] like Figure 2 As shown, the question-answer text data processing method includes steps S210 to S250.
[0026] In step S210, a question text sequence set is obtained.
[0027] Step S220: using a large language model to perform recognition processing on each question text sequence in the question text sequence set, and obtaining an answer text sequence corresponding to each question text sequence.
[0028] In step S230, a positive sample question text sequence corresponding to each question text sequence is determined based on each question text sequence and the answer text sequence corresponding to each question text sequence.
[0029] In step S240, based on the positive sample question text sequences corresponding to the respective question text sequences, corresponding negative sample question text sequences are generated.
[0030] In step S250, a vector retrieval model is trained based on the positive sample question text sequences and the negative sample question text sequences corresponding to the question text sequences in a loop iteration manner.
[0031] This method can more accurately understand and process the intent of user queries by using a large language model to recognize and process question text sequences and generate corresponding answer text sequences. By accurately matching questions and answers, the relevance and accuracy of retrieval results are improved. By generating positive sample question text sequences and corresponding negative sample question text sequences, richer and more diverse training data is provided for the vector retrieval model. This method not only enhances the model's adaptability to different types of questions, but also improves the model's robustness in practical applications. The use of a cyclic iterative training method enables the vector retrieval model to better optimize its performance after multiple rounds of training, effectively reducing the proportion of false retrievals and improving retrieval speed and efficiency.
[0032] In some embodiments of the present disclosure, the above-mentioned question text sequence set can be the basis for building and training a vector retrieval model. These question text sequence sets can be obtained in two main ways: one is a pre-configured question text sequence, and the other is a historical question text sequence obtained from a database. It can be defined and written in advance according to specific business needs. These question text sequences can cover typical or common problems in the business field to ensure that the model can identify and handle these high-frequency problems. The pre-configured question text sequence can accurately control the quality and scope of the training data, which helps the model to show higher efficiency and accuracy on specific problems. These sequences contain not only specific questions, but also background descriptions of the questions, relevant contexts, etc., which helps the model to understand the context and details of the questions more comprehensively. Historical question text sequences are collected from actual user interactions, such as query requests submitted by users during the use of services. These question text sequences reflect the real needs and expression habits of users and have high practical value. Using historical question text sequences allows the model to learn more diverse expressions and question types, and enhances the model's ability to handle unknown or rare problems. These question text sequences may also contain detailed background descriptions of the questions, enabling the model to understand and respond to the specific needs of users in a wider context. Through these sequences, the model not only learns how to accurately respond to specific queries, but also gradually optimizes its understanding of the background and context of the question, thereby providing more accurate and relevant search results in practical applications. This approach effectively improves the performance of the retrieval model, enabling it to better serve the specific needs of users.
[0033] Based on the above embodiments, the question text sequence in the above question text sequence set can also be constructed by using prompt mode or CoT mode. The prompt mode is a text generation method based on templates or samples, which guides the model to generate a specific type of text by designing some templates or examples. In the prompt mode, the user provides a question or task description, and then designs a template or example based on this description, so that the model generates the corresponding text sequence according to this prompt. The prompt mode is often used to generate tasks such as dialogue, text summary, translation, etc., and different prompts can be designed according to specific needs to guide the model to generate a specific type of text. CoT is a method for pairing code snippets with natural language text, and the model can identify the meaning of the code and generate a corresponding text description. In the CoT mode, each question text sequence is paired with a related code snippet, and the model understands the meaning of the code and generates a text description by learning these Code-Text Pairs. The CoT mode is often used for tasks such as code generation, code annotation, code summary, etc., and can help the model better understand and generate natural language text related to the code. Using the prompt mode or CoT mode to construct a question text sequence set can provide a more flexible way to guide the model to generate a specific type of text. By designing appropriate prompts or code-text pairs, the model can be customized to generate text sequences that meet specific needs. These methods can help the model learn knowledge of specific fields or tasks more quickly, thereby improving the efficiency and accuracy of text generation. By using the prompt method or CoT method to construct a set of question text sequences, the model can be provided with more instructive and customized training data, which helps the model better understand task requirements and generate corresponding text sequences. The application of this method has broad application prospects in the field of natural language processing and can promote the performance and effect of the model on specific tasks.
[0034] In some embodiments of the present disclosure, each question text sequence in a set of question text sequences is identified and processed by a large language model to obtain an answer text sequence corresponding to each question text sequence. For example, first, the question text sequence is input into the large language model. The large language model uses its pre-trained capabilities to understand the content and context of the question. The model analyzes the keywords, semantic structures, and any background information related to the question in the question. Based on the understanding of the question, the large language model generates one or more answer text sequences. In addition, the generated answer is not just a simple text, but also includes the detailed information and in-depth analysis required to answer the question. In this way, the application of the large language model significantly improves the processing quality of the question text sequence and the overall performance of the system, making the answer generation both accurate and highly relevant, thereby effectively supporting the training and optimization of the vector retrieval model.
[0035] In some embodiments of the present disclosure, according to each question text sequence and the answer text sequence corresponding to each question text sequence, determining the positive sample question text sequence corresponding to each question text sequence includes: performing a preset number of copy operations on the character string in each question text sequence based on a first preset formula to obtain the first question text sequence corresponding to each question text sequence; splicing the first question text sequence corresponding to each question text sequence and the answer text sequence corresponding to each question text sequence to obtain the positive sample question text sequence corresponding to each question text sequence. For example, first, perform a preset number of copy operations on each question text sequence. This is based on the first preset formula, and the specific content of the formula includes specifying the number of copies and the way of copying (for example, directly copying the entire character string or part of the character string). The purpose of the copy operation is to increase the richness of data and the amount of data during model training, thereby improving the model's recognition ability for similar questions. After the copy operation is performed, the result obtained is called the "first question text sequence". These sequences are similar to the original question text sequence in structure, and the frequency of their occurrence is increased by copying, which helps the model to better learn and understand the characteristics of these problems. Then the "first question text sequence" corresponding to each question text sequence is spliced with the corresponding answer text sequence. Splicing refers to merging two sequences into one sequence at the text level. The purpose of the splicing operation is to create a complete text sequence containing questions and answers, which is a complete information entity containing questions and their correct answers for the model. In this way, the model can learn the direct relationship between questions and their correct answers. Through the above-mentioned copying and splicing operations, the final text sequence generated is defined as a positive sample question text sequence. Positive samples usually refer to samples that correctly represent the patterns or answers that the model is expected to learn. These positive sample question text sequences will be used as input data when training vector retrieval models to help the model accurately learn how to retrieve the correct answers from question texts. In this way, the model's understanding of the relationship between questions and answers can be effectively enhanced, and the accuracy and efficiency of the model in practical applications can be improved. This strategy not only increases the diversity of data, but also enhances the practicality of data, so that the trained model can better adapt to the complex query needs of the real world.
[0036] Based on the above embodiment, the first preset formula is:
[0037] q new =concat({q old}*n,d);
[0038] Among them, q new is the positive sample question text sequence corresponding to each of the question text sequences, q old For each of the question text sequences, {q old}*n is the first question text sequence corresponding to each of the question text sequences, n is the number of times the string is copied, and d is the answer text sequence corresponding to each of the question text sequences. The original question can be expanded through the first preset formula, thereby improving the accuracy of the model in practical applications.
[0039] In some embodiments of the present disclosure, according to each question text sequence and the answer text sequence corresponding to each question text sequence, determining the positive sample question text sequence corresponding to each question text sequence includes: obtaining a preset delimiter; splicing each question text sequence, the delimiter, and the answer text sequence corresponding to each question text sequence based on a second preset formula to obtain the positive sample question text sequence corresponding to each question text sequence. For example, a positive sample question text sequence is generated by a specific splicing operation. This process ensures that the connection between the question and the corresponding answer is effectively learned by the model, thereby improving the retrieval accuracy. First, a preset delimiter needs to be selected. The function of the delimiter is to clearly distinguish the question text sequence and the answer text sequence during the splicing process. For example, the delimiter is [SEP]. According to the second preset formula, each question text sequence is connected to its corresponding answer text sequence through a delimiter. For example, if the question text sequence is "What is AI?", the answer text sequence is "AI is the abbreviation of artificial intelligence.", and the delimiter is "[SEP]", then the spliced positive sample question text sequence is "What is AI? [SEP] AI is the abbreviation of artificial intelligence.". The positive sample question text sequence processed in this way contains complete question and answer information, forming a single, coherent text entity. This is a training sample containing rich information for the model, which helps the model learn how to associate questions with correct answers. The generated positive sample question text sequence will be used as training data for the vector retrieval model. During the training process, the model will learn how to map from questions to answers, which is achieved by analyzing the semantic association between question text and answer text. Through the above steps, it can be ensured that each generated positive sample question text sequence reflects the direct connection between questions and answers with high quality, which is crucial for building an efficient and accurate vector retrieval system. This method not only improves the performance of the model, but also optimizes the user's query experience, greatly improving the relevance and accuracy of the answers.
[0040] Based on the above embodiment, the second preset formula is:
[0041] q new =conat(q old ,[SEP],d);
[0042] Among them, q new is the positive sample question text sequence corresponding to each of the question text sequences, q oldis each of the question text sequences, [SEP] is the separator, and d is the positive sample question text sequence corresponding to each of the question text sequences. The original question text sequence can be expanded by the second preset formula, so that the vector retrieval model trained with the expanded question text sequence is more accurate.
[0043] In some embodiments of the present disclosure, based on the positive sample question text sequence corresponding to each question text sequence, generating the corresponding negative sample question text sequence includes: calculating the similarity between the positive sample question text sequence corresponding to each question text sequence and the text sequence in the document index library; according to each similarity, determining the negative sample question text sequence corresponding to each positive sample question text sequence from the document index library. For example, first, it is necessary to select a suitable algorithm to calculate the similarity. In this example, the ANCE (Approximate Nearest Neighbor NegativeContrastive Estimation) algorithm is used. ANCE is an efficient algorithm for calculating the similarity between text sequences, and is particularly suitable for large-scale document collections. Using the ANCE algorithm, the similarity between the positive sample question text sequence corresponding to each question text sequence and other text sequences in the document index library is calculated. The document index library usually contains a large amount of text data, which may be related to or unrelated to the question text sequence. According to the calculated similarity, the text sequence with low similarity to the positive sample question text sequence is screened out from the document index library. These low-similarity text sequences are considered to be potential negative samples. The selection criteria are usually based on a similarity threshold, that is, text sequences with a similarity lower than a certain value are considered to be negative samples. The key to setting this threshold is to ensure that negative samples are clearly different from positive samples in content, so as to effectively train the model to distinguish between relevant and irrelevant queries. The generated negative sample question text sequence will be used for model training when performing negative sample comparative learning. In this training mechanism, the model not only learns to identify and answer questions related to positive samples, but also learns to identify and exclude irrelevant information related to negative samples. By using both positive and negative samples, the model can more accurately understand what type of query is related to a given answer, thereby providing more accurate and relevant results in practical applications. Through the above steps, negative sample question text sequences can be effectively generated, and the use of efficient algorithms such as ANCE can ensure the accuracy and efficiency of similarity calculation, making the model training process more scientific and reasonable.
[0044] Figure 3 FIG. 1 is a flow chart of another method for processing question and answer text data provided by an embodiment of the present disclosure. Figure 3 The above method further includes step S310 and step S320.
[0045] In step S310, the question text sequence currently input by the user is obtained.
[0046] In step S320, the question text sequence currently input by the user is processed by the vector retrieval model to obtain at least one answer text sequence corresponding to the question text sequence.
[0047] The method can obtain the question text sequence currently input by the user, and process the question text sequence currently input by the user through the vector retrieval model to obtain at least one answer text sequence corresponding to the question text sequence. The vector retrieval model can effectively process the user's query request and provide accurate and relevant answer text sequences, thereby greatly improving user experience and satisfaction. This technology has a wide range of applications, including but not limited to online customer service systems, intelligent assistants, knowledge base queries and other scenarios.
[0048] In some embodiments of the present disclosure, in the practical application of the vector retrieval model, the question text sequence input by the user is processed and the corresponding answer text sequence is retrieved. Specifically, the question text sequence currently input by the user is first captured. It is usually implemented through a user interface (such as a web page, a mobile application or a speech recognition system), and the user inputs the question into the system in text form. Before processing the input, the text usually needs to be preprocessed to improve the processing efficiency and accuracy of the model. The preprocessing step may include removing stop words, punctuation cleaning, text normalization (such as lowercase conversion), word segmentation, etc. The preprocessed text needs to be converted into a vector form for the vector retrieval model to process. This usually involves using word embedding (such as Word2Vec, GloVe or BERT, etc.) to convert the text into a numerical vector. Using the vector retrieval model, the vector (query vector) of the question text sequence input by the user is compared with the document vector stored in the model. The model will use similarity calculations (such as cosine similarity) to find the document vector closest to the query vector. This step is implemented by efficient algorithms such as the fast approximate nearest neighbor (ANN) algorithm to ensure that the most relevant answers are quickly found in a large data set. Based on the document vectors with the highest similarity, the corresponding answer text sequences will be retrieved from the database. These text sequences are stored in association with the document vectors in advance, and usually contain information that directly or indirectly answers the user's question. The multi-answer support system can provide more than one answer text sequence, but provide multiple relevant answers based on the similarity score, so that the user can choose the most satisfactory answer from them. These answer text sequences will be returned to the user interface and displayed to the user. The display method can be a text list, highlighting the most relevant answers, or other user-friendly formats.
[0049] Figure 4 It is a block diagram of a question and answer text data processing device provided by an embodiment of the present disclosure.
[0050] like Figure 4 As shown, the question and answer text data processing device 400 includes an acquisition module 410, a recognition processing module 420, a determination module 430, a generation module 440 and a training module 450.
[0051] Specifically, the acquisition module 410 is used to acquire a question text sequence set.
[0052] The recognition processing module 420 is used to perform recognition processing on each question text sequence in the question text sequence set by using a large language model to obtain an answer text sequence corresponding to each question text sequence.
[0053] The determination module 430 is used to determine the positive sample question text sequence corresponding to each question text sequence according to each question text sequence and the answer text sequence corresponding to each question text sequence.
[0054] The generating module 440 is used to generate a corresponding negative sample question text sequence based on the positive sample question text sequence corresponding to each question text sequence.
[0055] The training module 450 is used to train the vector retrieval model based on the positive sample question text sequences and the negative sample question text sequences corresponding to the question text sequences in a cyclic iteration manner.
[0056] The question-answer text data processing device 400 can more accurately understand and process the intention of the user's query by using a large language model to identify and process the question text sequence and generate the corresponding answer text sequence. By accurately matching questions and answers, the relevance and accuracy of the retrieval results are improved. By generating positive sample question text sequences and corresponding negative sample question text sequences, more abundant and diverse training data is provided for the vector retrieval model. This method not only enhances the model's adaptability to different types of questions, but also improves the robustness of the model in practical applications. The use of a cyclic iterative training method enables the vector retrieval model to better optimize its performance after multiple rounds of training, effectively reducing the proportion of erroneous retrievals and improving retrieval speed and efficiency.
[0057] In some embodiments of the present disclosure, the determination module 430 is configured to: perform a preset number of copy operations on the character strings in each of the question text sequences based on a first preset formula to obtain a first question text sequence corresponding to each of the question text sequences; and concatenate the first question text sequence corresponding to each of the question text sequences and the answer text sequence corresponding to each of the question text sequences to obtain a positive sample question text sequence corresponding to each of the question text sequences.
[0058] In some embodiments of the present disclosure, the first preset formula is:
[0059] q new =concat({q old}*n,d);
[0060] Among them, q new is the positive sample question text sequence corresponding to each question text sequence, q old For each question text sequence, {q old}*n is the first question text sequence corresponding to each question text sequence, n is the number of times the string is copied, and d is the answer text sequence corresponding to each question text sequence.
[0061] In some embodiments of the present disclosure, the determination module 430 is configured to: obtain a preset separator; and concatenate each of the question text sequences, the separator, and the answer text sequence corresponding to each of the question text sequences based on a second preset formula to obtain a positive sample question text sequence corresponding to each of the question text sequences.
[0062] In some embodiments of the present disclosure, the second preset formula is:
[0063] q new =conat(q old ,[SEP],d);
[0064] Among them, q new is the positive sample question text sequence corresponding to each question text sequence, q old is each question text sequence, [SEP] is the separator, and d is the positive sample question text sequence corresponding to each question text sequence.
[0065] In some embodiments of the present disclosure, the generation module 440 is configured to: calculate the similarity between the positive sample question text sequence corresponding to each question text sequence and the text sequence in the document index library; and determine the negative sample question text sequence corresponding to each positive sample question text sequence from the document index library based on the respective similarities.
[0066] In some embodiments of the present disclosure, the question and answer text data processing device 400 is also used to: obtain the question text sequence currently input by the user; process the question text sequence currently input by the user through the vector retrieval model to obtain at least one answer text sequence corresponding to the question text sequence.
[0067] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present disclosure. Figure 5As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules in the above-mentioned device embodiments are implemented.
[0068] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.
[0069] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0070] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.
[0071] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0072] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.
[0073] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.
Claims
1. A method for processing question and answer text data, characterized in that: include: Get the question text sequence set; Recognize each question text sequence in the question text sequence set by using a large language model to obtain an answer text sequence corresponding to each question text sequence; Determine a positive sample question text sequence corresponding to each question text sequence according to each question text sequence and an answer text sequence corresponding to each question text sequence; Based on the positive sample question text sequences corresponding to the respective question text sequences, generating corresponding negative sample question text sequences; A vector retrieval model is trained based on the positive sample question text sequences and the negative sample question text sequences corresponding to the question text sequences in a cyclic iteration manner.
2. The method according to claim 1, characterized in that Determining the positive sample question text sequence corresponding to each question text sequence according to each question text sequence and the answer text sequence corresponding to each question text sequence comprises: Based on a first preset formula, a preset number of copy operations are performed on the character strings in each of the question text sequences to obtain a first question text sequence corresponding to each of the question text sequences; The first question text sequences corresponding to each of the question text sequences and the answer text sequences corresponding to each of the question text sequences are concatenated to obtain positive sample question text sequences corresponding to each of the question text sequences.
3. The method according to claim 2, characterized in that The first preset formula is: q new =concat({q old }*n,d) ； Among them, q new is the positive sample question text sequence corresponding to each of the question text sequences, q old For each of the question text sequences, {q old }*n is the first question text sequence corresponding to each of the question text sequences, n is the number of times the character string is copied, and d is the answer text sequence corresponding to each of the question text sequences.
4. The method according to claim 1, characterized in that: Determining, according to each of the question text sequences and the answer text sequences corresponding to each of the question text sequences, a positive sample question text sequence corresponding to each of the question text sequences comprises: Get the preset delimiter; Based on a second preset formula, each of the question text sequences, the separators, and the answer text sequences corresponding to each of the question text sequences are concatenated to obtain a positive sample question text sequence corresponding to each of the question text sequences.
5. The method according to claim 4, characterized in that The second preset formula is: what new =try(q old ,[SEP],d); Among them, q new is the positive sample question text sequence corresponding to each of the question text sequences, q old is each of the question text sequences, [SEP] is the separator, and d is the positive sample question text sequence corresponding to each of the question text sequences.
6. The method according to claim 1, characterized in that Based on the positive sample question text sequences corresponding to the question text sequences, generating the corresponding negative sample question text sequences comprises: Calculating the similarity between the positive sample question text sequence corresponding to each of the question text sequences and the text sequence in the document index library; According to each similarity, a negative sample question text sequence corresponding to each positive sample question text sequence is determined from the document index library.
7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Get the question text sequence currently input by the user; The question text sequence currently input by the user is processed by the vector retrieval model to obtain at least one answer text sequence corresponding to the question text sequence.
8. A question and answer text data processing device, characterized in that: include: An acquisition module is used to obtain a set of question text sequences; A recognition processing module, used for performing recognition processing on each question text sequence in the question text sequence set by using a large language model to obtain an answer text sequence corresponding to each question text sequence; A determination module, used for determining a positive sample question text sequence corresponding to each of the question text sequences according to each of the question text sequences and the answer text sequence corresponding to each of the question text sequences; A generation module, used for generating corresponding negative sample question text sequences based on the positive sample question text sequences corresponding to each of the question text sequences; The training module is used to train the vector retrieval model based on the positive sample question text sequences and the negative sample question text sequences corresponding to the question text sequences in a cyclic iteration manner.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.